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Executive Summary 

Purpose and Description 

In 2001, the U.S. Department of Education and the Ministry of Education in China 
entered into a bilateral partnership to develop a technology-driven approach to foreign language 
learning that integrated gaming, immersion, voice recognition, problem-based learning tasks, and 
other features that made it a significant research and development pilot project for study. 

The purpose of this report is to describe the evaluation of a key outcome of this bilateral 
partnership, The Forgotten World. This application, which was developed by Coastline 
Community College and the Learning Games Network through a grant from the William and 
Flora Hewlett Foundation, uses the integrated approach set forth in the U.S. -China bilateral 
agreement. This program was implemented as a supplementary activity in middle school 
classrooms in western China to teach the English language and American culture to eighth-grade 
students. 

The evaluation was conducted in five treatment schools and five comparison schools 
during the 2009-10 school year and included approximately 3,500 students. The ten schools 
were identified by provincial authorities and then randomly selected for either the treatment or 
comparison groups. All eighth-grade classrooms were asked to participate; in two schools all 
classes participated, but in the other three schools only about half of the classes participated. 
Thus, there may be some selection bias in the sample. However, eighth-grade students in the 
treatment and comparison schools on average started the school year with similar initial levels of 
English-language performance. 

Teachers in the treatment schools attended a three-day training workshop in August 2009 
to become familiar with The Forgotten World and with the approach to teaching and learning on 
which the application was based. At the end of the training, each school developed a plan for 
implementing the program in the school. Schools were asked to begin using the program in 
supplementary classes in late September 2009 and to allow students access to The Forgotten 
World in school computer labs for at least one class period each week. 

Reading and listening tests were administered at the beginning and end of the year to 
eighth-grade students in schools participating in the study. In addition, questionnaires were 
completed by teachers and students in the schools participating in the study. Teachers were also 
asked to complete logs describing each class session in which the program was used. 

Several implementation issues encountered during the study significantly reduced the 
amount of time students could spend on The Forgotten World. The expected level of 
implementation was not realized until halfway through the study, after information technology 
infrastructure improvements were made at treatment schools. Even then, only about 70 percent 
of students reported that they used The Forgotten World at least once a week. This partial 
implementation during one semester represents a fairly limited treatment in comparison with 
other education studies of this type. 
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Findings 

Students in treatment schools with relatively low levels of initial proficiency, who scored 
in the bottom half of all students on the fall tests, evidenced larger gains in reading and listening 
than similar students in comparison schools. There were no differences in performance between 
treatment and comparison groups among students with higher levels of initial proficiency (those 
scoring in the upper half of all students on the fall test). Overall, students in treatment and 
comparison schools performed similarly on the proficiency tests administered at the beginning 
and end of the school year. 

Students in the treatment group also reported higher levels of motivation to learn English 
than students in the comparison schools. At the end of the year, 51 percent of students in the 
treatment schools reported that their motivation to learn English had improved, compared with 
40 percent of students in the comparison schools. Teachers in treatment schools also reported 
that students’ motivation improved. This is an important finding because motivation to learn a 
foreign language has been identified by Chinese language experts as a problem in Chinese 
schools and as a barrier to learning. 

Almost all of the teachers in the treatment schools (95 percent) who participated in the 
project reported that their use of The Forgotten World changed the way they think about 
teaching. The approach to language teaching and learning that forms the basis for The Forgotten 
World is learner-centered and interactive and presents students with problem-based tasks and 
immediate feedback. The teachers appreciated the fact that the program provided students with 
fun opportunities for practicing listening and speaking skills, with exposure to native English 
speakers. However, some teachers expressed concern that their primary responsibility was to 
prepare students for the English-language portion of the high school entrance exam that students 
take in ninth grade and that they did not feel that use of The Forgotten World would improve 
student performance on the exam. 

The statistically significant positive results of using The Forgotten World for the lower 
performing students along with the positive effects on student motivation is encouraging for an 
intervention that was only partially implemented. Further research could be conducted to help 
understand why the program produced these benefits, how to overcome implementation 
problems in schools lacking sophisticated computer technology, and why The Forgotten World 
did not produce positive benefits for the initially higher performers. 



x 



A Study of a Gaming Approach to English Language Learning for Middle School Students 




Chapter 1. Purpose and Background 



1.1 Purpose 

The Policy and Program Studies Service (PPSS), within the Office of Planning, 
Evaluation and Policy Development (OPEPD), conducted a study of the E-language Learning 
System named Open Language Learning Initiative for English (OLLI English). “ The program 
developed under this initiative. The Forgotten World, was designed to help students in middle 
schools in China learn English. This study was conducted in Chinese middle schools for two 
semesters from September 2009 to May 2010. 

The original online language learning project began as a joint partnership between the 
U.S. Department of Education (Department) and the Ministry of Education of the People’s 
Republic of China to create a highly innovative research and development (R&D) product to test 
the effectiveness of using online technology to deliver foreign language instruction within a real- 
world environment, consistent with integrating emerging theories of effective language learning 
practices. The original design called for the Department to develop 36 online lessons to teach the 
English language (reading, writing, speaking, and listening) and American culture to students in 
the People’s Republic of China. 

The Department did not complete building the online language learning system. Instead, 
the Department made existing materials that had been developed using Department funds, 
including scripts and source code, available to the public via the World Wide Web. 

Subsequently, the William and Flora Hewlett Foundation committed to building upon the 
Department’s work. In February 2007, the Hewlett Foundation awarded a grant to Coastline 
Community College to build a complete application to teach the English language and American 
culture to students in China. The application, The Forgotten World , is part of OLLI English, a 
program sponsored by the Hewlett Foundation to develop open education resources for English 
language learning. 

This report presents information about the implementation of The Forgotten World 
program in middle schools in western China during the 2009-10 school year. The original 
purpose of the study was to evaluate the efficacy of the language learning approach on 
motivation and student language proficiency (outcomes), believing that proof of concept of the 
approach would be applicable to students learning a second language in the United States. 
Because of difficulties in implementation, the efficacy of the program could not be thoroughly 
evaluated. Nevertheless, the report discusses challenges to implementation and user experiences 
with the language approach. 



1 E-Language Learning System (ELLS) is the name of the joint project that was launched as a result of the 
memorandum of understanding between the U.S. and China signed in 2002 

2 The Open Language Learning Initiative (OLLI) is the name given to the William and Llora Hewlett 
Loundation initiative to provide open source high quality language courseware at little or no cost to language 
learners worldwide 

3 The Lorgotten World is the name of the English language learning program designed for Chinese middle- 
school students that was developed by Coastline Community College and Learning Games Network with funding 
from the Hewlett Loundation 
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Chapter 1. Purpose and Background 



1.2 Background 

During President George W. Bush’s trip to Shanghai in October 2001 for the Asia-Pacific 
Economic Cooperation (APEC) Leader’s Meeting, the White House announced a series of 
e-leaming initiatives. One of these initiatives involved the development of an online language 
learning system applying advanced Web-based language instruction technologies. 

One year later, the U.S. secretary of education and the Chinese minister of education 
signed a bilateral memorandum of understanding for a joint research and development project, 
known as the U.S. -China E-Language Project. The Chinese minister noted that the project was 
the largest govemment-to-government education initiative since the normalization of relations 
between the two countries in 1979. 

The crux of the agreement was that each country, China and the United States, would 
develop online language learning programs for middle school students in the other country. The 
project aimed to create an innovative online language learning R&D program in English and 
Chinese that was 

• targeted at middle school American and Chinese students; 

• constructed as a joint collegial effort by top experts from China and the United States 
in the areas of language instruction, technology, evaluation, and dissemination; 

• delivered to students in China and the United States via the Internet; and 

• presented to students in an interactive gaming environment with speech recognition. 

The design parameters used to guide development of the programs included a number of 
innovative features that made it a significant R&D pilot for evaluation. The Chinese and 
English programs were to 

• incorporate the research-based benefits of teaching a foreign language within a real- 
world context to enhance the learning and transferability of communicative language 
skills; 

• use a model-practice-apply framework consistent with the theory of learning a foreign 
language by hearing it, practicing it, and using it; 

• provide for extensive practice essential to developing language fluency; 

• be learner-centered in that it builds systematically on prior vocabulary, word 
difficulty, and communicative complexity; 

• encourage language learning through creating a gaming environment within a real- 
world context that rewards students for making language progress; 

• present cultural cues as part of language instruction along with expanded 
opportunities to learn about particular features of the culture and way of life; 

• adopt a learning management system that integrates problem-based learning tasks 
throughout lessons to provide the learner with continuous feedback on progress and 
for instruction to reinforce weaknesses; and 

• include voice recognition to authenticate and provide feedback on the learner’s 
speech reproduction capability. 
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In addition to teaching language, the online language-learning systems were to promote the 
learning of each country’s culture, incorporating real-world illustrations that depict current 
lifestyles, including living habits, transportation, buildings, dress, and music. 

China completed development of an online program for U.S. middle school students, 
named Chengo, in 2004. Access to that program can be found at www.elanguage.cn . Chengo 
was evaluated through a series of focus groups conducted by the Department and documented in 
a previous report. 

The Department did not complete development of the online language-learning system. 
Instead, it decided to turn responsibility for development of the system over to the private sector. 
The Department released the following system components into the public domain by publishing 
them on the Department’s Web site: 

• the source code for the technology platform, minus proprietary software components 
for voice recognition and learning management; 

• thirty-six English-language scripts written by English as a second language (ESL) 
experts and aligned with English language learning content standards; 

• a review of the system by the contractor; and 

• findings from the market research request for information. 

When the Department released these materials into the public domain, it indicated that it 
would evaluate completed systems that carried out the original vision for the system. In February 
2007, the William and Flora Hewlett Foundation awarded a grant to Coastline Community 
College to build a completed system to teach the English language and American culture to 
students in China. In May 2007, the Department and the Hewlett Foundation convened a meeting 
of experts to discuss the development of the program as well as plans for an evaluation. 

This report is focused on the study of the English language and culture program 
developed for Chinese middle schools, with funding from the Hewlett Foundation, and does not 
include information on the Chinese language program developed for U.S. middle schools. 
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Chapter 2. Description of the Project and Early 
Development 

2.1 Overall Structure of the Project 

One of the unique features of this study is that it is not an evaluation of a preexisting 
program. Rather, it is a study of the implementation of a new program that was sponsored and 
developed by a number of partners. Exhibit 1 presents information on the major partners 
involved in the overall project in the United States and China. The Department entered into the 
agreement with the Chinese Ministry of Education to develop and study a new language learning 
approach. As stated earlier, the Department released the specifications for the program and 
agreed to fund an evaluation of the program meeting the major design requirements. Thus, when 
the Hewlett Foundation awarded a grant to Coastline Community College (and, later, Learning 
Games Network) to develop the program, the Department contracted with RTI International (and 
its partner in China, East China Normal University [ECNU]), to evaluate implementation of the 
approach (via online technology) in China. The Chinese Ministry of Education tasked the 
National Center for Education Technology to facilitate the U.S. team’s work in China by 
identifying sites for the study. 



Exhibit 1. Organizations Involved in the U.S.-China E-Language Project and Their Roles 



Organization 

U.S. Department of Education 



Ministry of Education of the 
People’s Republic of China 



William and Flora Hewlett 
Foundation 

Coastline Community College, 
Learning Games Network 

RTI International 

East China Normal University 

National Center for Educational 
Technology (NCET), 

Provincial Center for 
Education Technology (PCET) 
and the City Center for 
Education Technology (CCET) 

Northwest Normal University 
(China) 



Role 

Collaborated with the Ministry of Education on the overall design requirements 
of the U.S.-China E-Language Learning System. Funded contract to RTI 
International to study implementation. 

Collaborated with the U.S. Department of Education on the overall design of 
U.S.-China E-Language Learning System. Assigned responsibility for 
facilitating the project and identifying province for implementation to the 
National Center for Educational Technology. 

Funded the development of The Forgotten World as part of the Open Language 
Learning Initiative. 

Funded by a grant from the Hewlett Foundation, developed instructional 
software, The Forgotten World , using a video-gaming approach. Also 
responsible for teacher training and implementation in schools. 

Funded by the U.S. Department of Education to study use of the program in 
Chinese middle schools. 

Partnered with RTI to design and conduct a study of the program when 
implemented in Chinese middle schools (subcontractor to RTI). 

NCET was assigned by Ministry of Education to facilitate project activities in 
China and identify a province to implement The Forgotten World. PCET 
identified schools, served as primary contact with schools regarding 
participation, and was liaison with schools for arranging dates for teacher 
training and data collection. CCET provided hardware support and 
troubleshooting. 

Partnered with Learning Games Network to provide support for initial teacher 
training and ongoing support for teachers throughout the year. 



Exhibit reads: Multiple organizations and agencies in the United States and China partnered to develop, implement, and evaluate 
new software designed to help Chinese middle school students learn English. 

Source: The U.S.-China E-Language Project. 
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The software development team, led by Coastline Community College and Learning 
Games Network, was responsible for implementation and teacher training and support. Locally, 
they partnered with faculty at Northwest Normal University for teacher support and the City 
Center for Education Technology (CCET) for technology support. Thus references in this report 
to the software and implementation team refer to Coastline Community College, Learning 
Games Network, Northwest Normal University, and CCET, all funded by the Hewlett 
Foundation. References to the evaluation team refer to RTI International and ECNU. Because 
the Department funded only the evaluation effort, that is the primary focus of this report. 

2.2 Initial Planning 

An Evaluation Planning Group (EPG) meeting was held in May 2007 in Beijing as part 
of a larger Technical Working Group (TWG) meeting to discuss the development of the program 
and the design of the evaluation. Experts in English language learning, educational evaluation, 
and educational gaming from China and the United States attended the meeting. At the meeting, 
officials from the Chinese Ministry of Education stated that they would like to implement the 
program in the middle schools in the western provinces of China because qualified English 
teachers are less available in those areas than in eastern and central China. At the time of the 
meeting, a Modem Distance Education Project was planned that would equip 35,000 middle 
schools in western China with new computers and Internet access, and the software development 
team was encouraged to design an application that would make full use of this advanced 
technology. 

Chinese scholars in the group expressed concern that middle schools might be reluctant 
to devote instructional time to experimental teaching methods like The Forgotten World program 
because of the importance of student performance on high-stakes entrance exams. Students in 
ninth grade must pass entrance exams to be accepted into high school, and English is one of the 
subjects included on the exam. Experts were also skeptical about the amount of time parents 
would allow students to use any type of gaming program at home, even if it was instructional 
gaming. Thus, the group consensus was that the system be designed for supplemental instruction, 
to be used in addition to the regular English classes. Because of the emphasis on high school 
entrance exams in ninth grade, the group suggested that the program be developed for seventh- 
or eighth-grade students and suggested that participating schools be selected from a single 
district to limit variation based on differing curriculum, textbooks, and length of English 
language instruction. 

The EPG recommended that a random assignment experiment be conducted with 25- 
35 treatment schools and an equal number of comparison schools to answer the research 
question: Do students exposed to the English program demonstrate increases in proficiency when 
compared with students without access to the program, given the same curriculum and under 
conditions that allow reasonable access to The Forgotten World? The group suggested that a mix 
of performance measures be used, including existing assessments, performance on district 
assessments, and specially designed assessments, if possible. Measurement of student motivation 
and attitudes was also recommended. Minutes of the EPG meeting are provided in Appendix A. 

2.3 Overview of The Forgotten World 

The Forgotten World was developed by Coastline Community College (a major provider 
of higher education distance learning courses) and the Learning Games Network with funding 
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provided by the William and Flora Hewlett Foundation. Following the guidelines initially set 
forth by the Department (described in Section 1.2) and the recommendations of the TWG, the 
program was designed for the target population of seventh- and eighth-grade students with prior 
exposure to English but limited vocabulary. The initial grant funded the development of the first 
12 episodes of a potential 36-episode series. The program was designed to be delivered to 
schools via the Internet for use in school computer labs. Students could also access the program 
from home computers using their assigned login credentials. A detailed description of the 
program can be found in Appendix B, a Quick Reference Guide to The Forgotten World. 

One unique feature of the program is that it has been designed as an open education 
resource that can be freely used, adapted, and repurposed by others, if desired. The underlying 
software is structured so that it can be easily modified for English language learners whose 
native language is not Chinese. For example, the components of the program (the story and 
activities) remain the same, but the dictionary can be changed from English-Mandarin to 
English-Spanish, enabling Spanish-speaking students learning English to use the program. The 
combination of open source code, video gaming, and easily modifiable dictionaries has the 
potential for creating derivative products for a wide range of audiences. 

The Forgotten World is presented in a Web browser. Learners are required to complete a 
simple registration process before becoming immersed in a comic-book adventure where the 
learner is one of the characters. A basic vocabulary assessment is woven into the first episode. 
The assessment is presented as a basic puzzle game in which the application gathers information 
about the learner’s proficiency of a vocabulary set drawn from approximately 500 words 
representing the baseline awareness expectation of the low-novice learner for whom the program 
has been designed. 4 The story then continues to unfold through an interactive graphic-novel 
format across 12 individual episodes, including the following: 

• Comic book story panels — Between 15 and 20 screens per episode communicate 
the story through full-color art, interactive dialogue bubbles, and audio clips; the story 
panels provide the learner with his or her first exposure to the vocabulary words and 
pronunciations. 

• Activities — Clickable activities and games enable the learner to interact with specific 
screens to support reading and listening skills development. 

• Games — Nested within each episode are two Flash-based games designed to support 
reading, listening, and speaking skills development. The games use vocabulary and 
simple phrases introduced in that or previous episodes as the “learning objects” to 
support learning goals; after a series of three to four episodes, games provide an 
opportunity to create formative evaluations from larger sets of content. 

• Resources — A personal digital assistant (PDA) with scoring and progress reporting 
function, episode log, dictionary, and games is displayed on screen; the PDA is 
designed to enable the learner to break out of the linear graphic-novel presentation 
and access previously viewed episodes and previously played games. 



4 The American Council on the Teaching of Foreign Languages (ACTFL) has developed a set of proficiency levels 
that include novice, intermediate, advanced, and superior. Within the novice category, the low -novice reading level 
is described as “able occasionally to identify isolated words and/or major phrases when strongly supported by 
context” (https://www.languagetesting.eom/assessments_academic.cfm#). 
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• Progress Reports — Scores are provided to the learner through the onscreen PDA 
interface. Teachers and parents have access to separate evaluation reports that convert 
the learner’s score to information about his or her achievement of specific learning 
goals. 

The learner has the freedom to navigate the screens within each episode, moving forward 
and backward at his or her own pace. Within the context of the narrative, games are launched to 
create opportunities for the learner to rehearse and apply vocabulary he or she has been exposed 
to or mastered in previous episodes plus newly introduced words and phrases. 

Because the level of each vocabulary word is coded within the application and presented 
at a specific time in an episode, the application adapts to the learner’s progress and performance. 
Following the learner’s completion of the preliminary evaluation, the application captures a set 
of variables that define the user’s starting level. From there, words and phrases that are 
appropriate to that level of performance as part of each episode populate the games in that 
episode. Depending on the learner’s achievement and scoring, the system can either raise the bar 
and introduce more difficult words and phrases or remediate by repopulating the games with the 
learning objects to reinforce the basics. The application is designed both to move the learner 
within a specific level across episodes so he or she can complete the application and to advance 
the learner’s performance through increased difficulty as he or she demonstrates proficiency and 
mastery of learning goals. 

2.4 Pilot Study 

Instrument identification.The EPG identified several potential tests that might be used 
in the evaluation. The list below contains the major tests reviewed for possible use: 

• The Test of English for International Communication (TOEIC) Bridge is a product of 
ETS and was developed to measure emerging English language competencies among 
beginning and intermediate learners. The paper-and-pencil tests take approximately 60 
minutes each. The tests were primarily designed for business use and contain many 
practical items. 

• Maculaitis Assessment of Competencies II, Test of English Language Proficiency 
(Touchstone Applied Science Associates), MAC II, is used by some states to measure 
English language proficiency among ESL students. 

• IDEA 1PT (Ballard and Tighe) is another test used by some states to measure English 
language proficiency among ESL students. 

• Stanford English Language Proficiency (SELP) Test , published by Harcourt Assessment, 
Inc., is another exam used by states to measure English language proficiency among ESL 
students. 

• Cambridge Young Learners Exam (YLE) is a criterion-referenced exam designed for use 
with elementary school aged children. A member of the EPG stated that this exam is used 
and respected in China. 
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• ACTFL Oral Proficiency Inten’iew (OPI). This is a new proficiency exam developed by 
the American Council on the Teaching of Foreign Languages (ACTFL) for 
administration via phone or Internet. The Internet connectivity of participating schools 
may be an issue for this methodology. 

These assessments were reviewed to determine the level of English assessed, skill areas assessed, 
potential to measure growth, time required for administration, special requirements for 
administration, appropriateness for age group, cultural appropriateness, cost, option to adapt the 
instrument, and vocabulary overlap with the English textbook used in the pilot school and The 
Forgotten World. 

Based on the preliminary review, the TOEIC Bridge was rejected as being inappropriate 
for this age group. The publishers of the SELP were unwilling to work with organizations not 
affiliated with school districts, and the Cambridge tests were criterion refererenced and not 
designed to yield gain scores. The ACTFL test was considered, but because of concern about the 
fact the test was still in development, it was not selected for this study. The assessments that 
were chosen for testing in Gansu included the IPT reading, writing, and combined 
listening/speaking tests, and the MAC II reading, writing, and listening tests (see Appendix C for 
more detail). 

December advance visit. In November 2008, the Ministry of Education identified a 
province, the western province of Gansu, and two pilot schools in that province willing to 
participate in the project. A meeting in the provincial capital was scheduled for December 9-11, 
2008. Representatives from the U.S. Department of Education, Coastline Community College, 
Learning Games Network, RTI International, and East China Normal University met with 
officials from NCET, the provincial Education Bureau, PCET, and principals and teachers of the 
two schools that volunteered to participate. 

The purpose of the visit was to explain the overall project, provide demonstrations to 
teachers, and gather feedback from the schools. Arrangements were also made to administer 
English language tests to a handful of students to provide preliminary information on the level of 
students’ English and the appropriateness of tests identified as potentially suitable for the study. 

Upon returning to Beijing en route to the United States, the study team met at the 
Ministry of Education to discuss arrangements for the pilot study. The study team expressed 
some concern about the level of technology observed in the labs in the two schools. The director 
of NCET noted that funding provided by NCET to provinces and schools was not restricted to 
the purchase of certain types of hardware configurations. Localities could choose how to spend 
the money they were allocated and thus lab configurations varied across schools. And given the 
recent earthquake in Sichuan, monies were prioritized for the schools in the affected areas, and 
additional funds to upgrade computer labs for this study were unavailable. 

A decision was made to conduct the pilot and main study in Gansu. The level of English 
instruction in the middle school was consistent with the program, and provincial officials, 
headmasters, and teachers were enthusiastic and willing to participate. The software 
development and implementation team decided that it could provide limited upgrades, such as 
increased memory for machines and headsets, that would enable the labs to be used for the study. 

Following the visit, extensive analysis was conducted comparing the vocabulary used in 
the tests with the vocabulary in the textbooks used by seventh- and eighth-grade students, Go 
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For It. In addition, the appropriateness of individual items was examined by identifying items 
that few students answered correctly, and examining whether the vocabulary was covered in the 
textbooks used, or if perhaps some element of the question was culturally inappropriate. The 
results of this analysis suggested that the MAC II listening test should be piloted in April, along 
with the IPT reading and writing tests. More information on these analyses can be found in 
Appendix D. 

April-June pilot. The main pilot activities occurred from April through June 2009; a 
complete list of activities for the pilot and main study is provided in Exhibit 2. In the initial 
statement of work for this study, a yearlong pilot was scheduled for 2008-09. However, given 
delays in the development of the program, the pilot consisted of a three-month implementation of 
the program in two pilot schools. During the April-June pilot, The Forgotten World was 
implemented in seventh- and eighth-grade classrooms in both schools. 

The purpose of the pilot data collection in April was to determine: 

• whether the software was ready for a full-scale implementation in the fall, 

• whether the English language reading, listening, and writing tests identified in 
December were appropriate for use with this population, and 

• the suitability of beginning-of-year questionnaires for students and teachers. 

Assessment instruments and questionnaires were tested in two pilot schools in the 
province in spring 2009; more than 1,800 students were tested. The IPT Reading and IPT 
Writing assessments were used, along with the Mac II Listening assessment. The IPT Reading 
assessment was used without modification; the other two assessments were adapted based on 
preliminary work in China in December. For the writing test, some of the picture prompts were 
substituted using pictures from other IPT tests that were more culturally appropriate for Chinese 
middle school students. The listening test directions were modified so that the directions were 
delivered to students in Chinese, and the answer form was modified to make it more 
understandable to students. All of the listening instructions were professionally recorded to 
ensure comparability across sites. 

One of the major purposes of the pilot was to make sure that the tests were suitable for 
use among this population and that the modifications made to the tests did not alter the 
equivalency of the pre- and post-test forms. The tests were administered at two pilot schools to 
seventh- and eighth-grade students. Because the delays in developing the program meant that a 
full year pilot was impossible, end-of-year seventh-grade students were tested to represent scores 
that might be expected from beginning eighth-grade students. It was not possible to measure 
growth in proficiency for individual students. 
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Exhibit 2. Schedule of Study Activities 



Date Activity Conducted by: 



Fall 2008 


Identification of potential tests 


RTI International 


December 2008 


Meeting with provincial officials and pre-pilot 
testing 


U.S. Department of Education, RTI 
International, East China Normal 
University (ECNU) 


April 2009 


1 .5 days of training of teachers in pilot 
schools 


Coastline Community College and Learning 
Games Network, Professor Zhang 
Lianzhong 


April 2009 


Pilot of tests and pretest questionnaires, all 
seventh- and eighth-grade students in 
pilot schools; meeting with provincial 
officials to outline requirements for 
2009-10 study 


RTI and ECNU train test administrators to 
administer pilot tests and questionnaires 
to seventh- and eighth-grade students in 
two pilot schools (30+ classrooms) 


June 2009 


Data entry and scoring 


ECNU; RTI conducted reliability scoring 


June 30— July 3, 2009 


End-of-pilot posttest questionnaires 
administered in six classrooms; focus 
groups and interviews with subset of 
students, teachers, principal, support 
personnel, and provincial center 


ECNU team: Professors Zhu Yiming, Wang 
Jiqing, and Dr. Yang Guangfu of ECNU 


June 25, 2009 


Schools selected for inclusion in 2009-10 
study 


Provincial Center for Education Technology 
(PCET) 


August 2-5, 2009 


2.5 days of training for teachers; computer 
labs at participating schools visited and 
computers checked 


U.S. Department of Education, Ministry of 
Education (People’s Republic of China), 
William and Flora Hewlett Foundation, 
Coastline Community College, Learning 
Games Network, Northwest Normal 
University, RTI, ECNU, invited guest 


August-September 

2009 

September 24, 2009 


Implementation plans negotiated with 
schools 

Use of The Forgotten l/Vor/c/ begins in 
classrooms 


ECNU 


September 28- 

October 13, 2009 
(No testing first week 


Tests and questionnaires administered 


ECNU and RTI train test administrators and 
administer tests in 1 0 schools 
(95 eighth-grade classrooms) 


of October; 
national holiday) 


October 12-13, 2009 
December 2009 
January 2010 


Observation of The Forgotten World classes 
in two schools 

Dell agrees to donate a new server for each 
school 

New servers delivered 


RTI 


January 2010 
January-February, 


Site visit 
Schools closed 


ECNU 


2010 


March 2010 


New servers installed and refresher training 
launched 


Learning Games Network, PCET, Northwest 
Normal University, ECNU 


June 20-26, 2010 


End-of-year tests and questionnaires 
administered 


ECNU 


July 2010 


Data entry and scoring 


ECNU; data quality checks at RTI 



Source: The U.S. -China E-Language Project. 



Results for each of the major tests are presented in Appendix D. For both the reading and 
listening tests, the two forms (pretest and posttest) were roughly equivalent. There were no 
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statistically significant differences in the raw scores for the two reading test forms. A significant 
difference was found for the mean raw scores on the two forms of the listening test; however, 
conversations with the publisher and published manuals show that the equivalence between 
forms is achieved through adjustments in transforming the raw scores into standard scale scores. 
Also, comparing results for seventh- and eighth-grade students, eighth-grade students 
consistently performed better than seventh-grade students on both the reading and listening tests, 
and the differences were statistically significant. The writing test showed the weakest results. A 
large number of students left the writing test blank; in focus groups, students reported that they 
were unfamiliar with the test format (using picture prompts) or had little or no experience writing 
English or both. Because of this, and the fact that proficiency in writing was not a goal for the 
e-language approach, a decision was made to drop the writing test from the final battery. 

Test results were also validated by comparing them with scores on district tests provided 
by the schools. As expected, students scoring well on district exams performed well on the 
proficiency assessments. The scores on school exams were more correlated with reading and 
writing assessments, and less correlated with the listening assessment. This finding was 
consistent with the rationale for the study; teachers and students in rural areas have little 
opportunity to interact with native English speakers; consequently, their listening (and speaking) 
skills are frequently weaker than their reading skills. 

In June, end-of-year questionnaires for students and teachers were administered and a 
series of focus groups and interviews were conducted with administrators, teachers and students 
to gather feedback on their experiences with The Forgotten World. Results of these interviews 
and focus groups indicated the following: 

• Both pilot schools valued the experience and asked to continue using The Forgotten 
World and to be included in the full-scale evaluation. 

• Participants complained about the slowness of the software. As a result, the software 
was modified and compressed by the software developers. 

• Teachers stated that they needed more training. As a result, a three-day training 
workshop was developed for the full-scale evaluation. Periodic meetings and 
trainings throughout the year for participating teachers were also scheduled for 
teachers participating in the 2009-10 study. 

• Teachers indicated that they would like more clarity on goals and expectations. Plans 
for the August training were modified to provide more information for teachers and 
principals, including clear expectations for the use of the program. 

• Students expressed concern about the dictionary, and some were concerned that the 
speech used was too fast for some to comprehend. The software development team 
took this matter under consideration, because experts had suggested limiting the 
amount of native language available and using conversational speed consistent with 
that of native speakers. 

• When asked about the writing tests, students said that they were unfamiliar with 
writing English and also unfamiliar with picture prompts. As a result, a decision was 
made to drop the writing test from the full-scale study. 
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The original intent of this project was to conduct a random assignment experiment to 
evaluate the impact of this new approach to language learning on student proficiency in English. 
Numerous factors intervened to make this design unfeasible, primarily the difficulty in 
identifying schools that had computer labs that met the minimum requirements for the study. The 
research questions and study design were modified as a result and are described below. 

3.1 Research Questions 

In June 2009, the provincial authorities determined that there were not enough schools 
that met the technological requirements for participation in the project. They were able to 
identify 10 schools, with a combined enrollment of approximately 6,000 eighth-grade students, 
that met project requirements: 

• schools with computer labs capable of running The Forgotten World ; 

• schools committed to allowing eighth-grade students access to the computer lab at 
least one hour per week; 

• schools using Go For It, the English textbook used in the pilot schools; 

• schools with teachers able to attend training in early August 2009; and 

• schools willing to allow testing in September 2009 and June 2010. 

While the number of schools was lower than requested, a decision was made that the 
study would continue and would use the 10 schools that met the requirements (A move to a 
different province would delay the study by one year or more.) An RTI statistician randomly 
assigned five schools to the treatment group and five schools to the comparison group. It was 
acknowledged that this design did not allow the statistical precision required of a hypothesis 
testing study but would be a solid foundation for a hypothesis-generating study. Using a smaller 
set of schools would allow more support for implementation, and retaining some schools for 
comparison would help provide a frame of reference for the revised research questions: 

• What challenges do schools face in implementing The Forgotten Worldl 

• Do findings suggest that students who use The Forgotten World frequently improve 
their English proficiency more than students who use The Forgotten World less 
frequently or not at all? 

• Does The Forgotten World increase student motivation to learn English? 

• What are students’ and teachers’ attitudes toward the use of The Forgotten Worldl 

3.2 Study Design 

Exhibit 3 provides information on the 10 schools that were identified by provincial 
officials as meeting the criteria for participation. The evaluation team discussed options for the 
assignment of schools and teachers with the Department. Concern about the viral nature of some 
gaming applications and the potential for “contamination” across classes indicated that random 
assignment should occur at the school level, rather than the classroom level. Thus, the schools 
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were randomly divided into two groups by RTI; the schools selected to use The Forgotten World 
are highlighted. All eighth-grade classes in the highlighted schools were asked to participate, and 
eighth-grade English teachers were invited to attend training in early August 2009. However, not 
all teachers and classrooms in treatment schools participated in the study, due to the capacity of 
labs, scheduling issues, and the voluntary nature of the study. In two schools all eighth-grade 
classrooms participated, and in the remaining three about half of the eighth-grade classrooms 
participated; thus about 64 percent of classrooms in treatment schools participated (see Section 
4.2 for more information about participating classes). Participating classrooms in the treatment 
schools were identified through implementation plans prepared by the schools at the end of the 
August training and then updated in the fall when implementation began. Classrooms were 
selected for inclusion by the schools based on teachers’ willingness to participate and scheduling 
of computer labs. Teacher logs were collected monthly from participating classes to confirm 
active participation, and site visits were made to observe classes. No classrooms in treatment 
schools dropped out of the study. All eighth-grade classrooms in comparison schools were 
included in the fall data collection; a random sample of all classrooms was tested at the end of 
the year. 



Exhibit 3. Characteristics of Participating Middle Schools 







Number of teachers 




Number of classes 




Number of 
computer 
rooms 


Number of 
computers 
for students 


School type 


Enrollment 


Total 


English 

teachers 


Total 


Grade 

7 


Grade 

8 


Grade 

9 


City 


2,300 


168 


23 


40 


12 


12 


16 


2 


150 


City 


1,700 


130 


16 


25 


8 


8 


9 


4 


172 


City 


1,400 


90 


15 


24 


8 


8 


8 


1 


80 


City 


635 


66 


13 


16 


6 


5 


5 


2 


131 


Rural 


1,300 


126 


17 


24 


8 


8 


8 


2 


135 


Rural 


1,300 


36 


8 


19 


8 


6 


5 


1 


50 


Rural 


2,701 


117 


19 


36 


13 


13 


10 


2 


61 


Rural 


2,550 


114 


18 


33 


12 


12 


9 


2 


96 


Rural 


1,990 


94 


16 


30 


12 


12 


6 


1 


35 


Rural 


1,203 


52 


10 


18 


8 


10 


0 


2 


96 



Exhibit reads: Middle schools in China chosen for inclusion in the project have large enrollments and large class sizes. 
Source: The U.S. -China E-Language Project. 



The middle schools were located in two areas within the province. As shown in Exhibit 3, 
most of the middle schools selected for inclusion in the study enroll large numbers of students. 
The smallest school enrolled 645 students; the remaining 9 schools enrolled between 1,200 and 
2,700 middle school students. All schools appeared to have the requisite computer equipment, 
although later it was found that in many schools the Internet connectivity was unstable and the 
average class size for middle school English classes exceeded the capacity of the computer labs. 

Instruments tested or developed during the pilot were modified slightly for use in the 
2009-10 study and included the following: 

• the IPT 2004 Reading Test — Forms 1C and ID for use at the beginning and the end 
of the school year (Amori, Dalton, and Tighe 2008); 
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• the MAC II Listening Test — Forms A2 and B2 (Maculaitis 2001) for use at the 
beginning and the end of the school year (minor modifications were made to Form 
A2, based on pilot results); 

• student and teacher questionnaires for the beginning and the end of the school year; 
and 

• weekly teacher logs. 

No speaking test was used because the logistics required to implement one-on-one testing 
in English would have required additional resources. Also, the final version of The Forgotten 
World did not incorporate speech recognition, although students using it were asked to produce 
and practice speech. 

An effort was made to limit the amount of time required for students to complete the tests 
and questionnaires. The total time required in each class was just under two hours: 10 minutes to 
introduce the study and distribute the forms, 60 minutes for the reading test, 30 minutes for the 
listening test, 15 minutes for the questionnaire, and 5 minutes to collect the forms. 

In addition to collecting the quantitative data, study staff made three site visits to 
participating schools. The first site visit, in October 2009, following the fall data collection, was 
made by a researcher from RTI to identify any issues with early implementation. Two interim 
site visits were made by staff from ECNU to observe implementation in the schools in winter and 
spring 2010. 

Teachers were also asked to complete logs describing each class session. Research has 
shown that teachers are accurate reporters of classroom activities (Rowan and Correnti 2009). 
These logs were submitted monthly to ECNU, and feedback was provided to teachers as needed. 
See Appendix E for information on the revised evaluation plan. 

3.3 Data Collection 

Data collection was conducted in the 10 participating middle schools described in 
Section 3.2. Data were collected in late September 2009 and in late June 2010. Chinese test 
administrators (many from the local university) were trained to distribute consent forms, check 
the class rosters, administer the test, and collect tests and questionnaires from students and 
teachers (see Appendix F for the fall training manual and questionnaires and Appendix G for the 
end-of-year training manual and questionnaires). RTI staff were present and supervised the 
September data collection activities. 

In both the September 2009 and June 2010 data collections, students were read a 
statement about the voluntary nature of the study and then were asked to complete the reading 
test and listening test and to fill out a paper-and-pencil student questionnaire. The listening test 
was administered using a tape recorder or, in a few cases, the school’s public address system. 
(Administration using a tape recorder is authorized by the test publisher.) All participating 
teachers were asked to fill out a paper-and-pencil teacher questionnaire. 

All eighth-grade classrooms in treatment schools were included in the fall 2009 testing 
because it was unclear as to which teachers and classes would participate; all eighth-grade 
classes in comparison schools were also tested in the fall. The total sample size for the fall 2009 
data collection was 6,073 students within 95 classrooms, including 3,102 students from five 
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treatment schools and 2,971 students from five comparison schools. The response rate for the fall 
was 98.6 percent in the treatment schools and 99.1 percent in the comparison schools. A student 
was considered a respondent if he or she completed at least one of the assessments. 

In the June 2010 post-test, only eighth-grade classrooms that used The Forgotten World 
were tested in the treatment schools, along with an approximately equal sample of classrooms 
from the comparison schools. In the fall, there were 3,938 students in 62 classrooms that were re- 
tested at the end of the year: 1,967 students from treatment schools and 1,971 students from 
comparison schools. In all, 3,640 of these students completed at least one assessment at the end 
of the year, for an overall response rate of 92.4 percent; within the treatment schools, 93.4 
percent of students in participating classes completed the spring assessment compared to 91.4 
percent of students in the comparison schools). 

The smaller number of students tested in June 2010 does not represent attrition; rather, a 
decision was made to test all students in treatment schools in fall 2009 because not all schools 
had provided final implementation plans, and the data collection contractor did not know exactly 
which classrooms would participate. The testing materials were shipped approximately one 
month before data collection, and at that time the participation status of some schools was 
unclear. 

Teachers of eighth-grade English were also asked to complete questionnaires. All of the 
48 teachers identified in the fall (24 each in the treatment and comparison schools) completed the 
questionnaires for a 100 percent response rate. In the spring, the 22 participating teachers in the 
treatment schools and 26 teachers in the comparison schools (including new teachers) completed 
questionnaires, for a 100 percent response rate. 

3.4 Data Processing 

The final dataset used for most of the analysis includes only those records that appear on 
both the September 2009 and June 2010 files, a total of 3,640 records. The final dataset includes 
test IDs, student test scores, records of the amount of time spent using The Forgotten World (as 
recorded by the software developer on U.S. servers), student questionnaire response data, and 
teacher questionnaire response data. RTI provided the data entry templates to ECNU to 
implement the initial data entry. RTI staff performed quality comparison checks to ensure the 
data quality. 

The student test ID is a nine-digit number, which is composed of a three-digit school ID, 
a three-digit classroom ID, and a three-digit student ID. The teacher questionnaire ID is a 
six-digit number including a three-digit school ID and a three-digit teacher ID. The teacher 
record also includes the class IDs for the classes taught by that teacher. The IPT1 reading test has 
24 questions, and the MAC II listening test has 53 questions. Data from the beginning and end- 
of-year questionnaires are also included in the data files for teachers and students. 
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3.5 Data Analysis and Statistical Methods 

Data analysis was conducted using SAS. Because of the relatively small number of 
schools included in the study, no attempt was made to estimate complex models. Because the 
purpose of the study was primarily descriptive, /-tests were used to compare results across 
groups. Also, because of the clustering of students within schools, design-corrected standard 
errors were computed and used for significance testing, using the SAS Surveymeans procedure. 

In this report, no results are statistically significant unless specifically stated. 
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4.1 Teacher Training 

The workshop for training teachers was held August 3-6, 2009, on the campus of a local 
university in Gansu province. Headmasters, teachers, and school computer lab directors were 
invited to attend. Forty-four participants completed the training; four of the schools sent all 
eighth-grade English teachers, while one school sent one teacher (a supplemental training was 
held at this school in October). Participants from distant schools and the project team were 
housed in the campus conference center. Large sessions were held in the conference center, and 
hands-on practice sessions were held in a computer lab on campus. Sessions presented by 
persons from the United States were translated into Chinese. 

The agenda for the training session is presented in Exhibit 4. Officials from the U.S. 
Department of Education and the Ministry of Education of the People’s Republic of China 
described the history and purpose of the U.S. -China E-Language Learning System and 
emphasized the importance of the schools’ participation in the project. Experts in teaching and 
learning were also asked to speak to the group about the program’s significance, emphasizing 
that the approach to learning embedded in The Forgotten World differed from the traditional 
approach typically used in middle schools. 

Based on the results of the pilot study, the length of the training was extended, and 
teachers were given more hands-on practice with the application. Sessions also included 
demonstrations of how the program had been used in classes during the pilot, a review of the 
teacher’s guide, and planning time for teachers to develop lesson plans based on The Forgotten 
World. The Ministry of Education and the Provincial Education Bureau presented the 
expectations for program implementation and discussed the practical requirements of the project. 

On the final day of training, teachers and headmasters were asked to meet with others 
from their school to develop an action plan for implementing the program at their school. When 
the teachers presented their school action plans, it became clear that there were challenges and 
that some schools might be unable to meet the program implementation requirements. One 
school’s representative stated that only one eighth-grade class would participate; another 
school’s representative said that only one teacher would implement The Forgotten World. These 
schools were reminded that the minimum requirement for participation was half of the eighth- 
grade classes. In two schools, the size of the average class was much larger than the size of the 
computer labs, and not all students would be able to use The Forgotten World during each class 
period. These issues were discussed, and a decision was made to follow up with these schools 
and the Ministry of Education to ensure that students had adequate exposure to The Forgotten 
World. 

At the end of the training, participants were asked to complete a questionnaire about the 
training workshop. Forty-four questionnaires were completed. Participants were asked to rate 
major components of the training on a five-point scale, where 1 was “not good at all” and 5 was 
“very good.” Mean scores for each component are presented in Exhibit 5. In general, the training 
received ratings of “good” or better with one exception. Participants’ rating of their own level of 
independent use was rated between “average” and “good.” 
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Exhibit 4. Agenda for Teacher Training 

The Forgotten World 
Teacher Training Workshop 

Monday, August 3-Thursday, August 6, 2009 


Monday, August 3 


1 :30— 5:00 p.m. 


Preparatory Meeting for Coastline Community College, Learning Games 
Network, Gansu Provincial Center for Education Technology, National Center 
for Educational Technology, and Northwest Normal University Teams 


Tuesday, August 4 


8:30-9:30 a.m. 


Welcome and Introduction 


9:30 a.m.-10:00 a.m. 


Demonstration of The Forgotten World 


10:15 a.m.-12:00 p.m. 


Practice Session 1 in Computer Lab 


12:00-2:30 p.m. 


Lunch and Break 


2:30-3:45 p.m. 


Overview of Teacher’s Guide 


4:00-6:00 p.m. 


Keynote and Discussion 

Playing Games and English Teaching: How to understand and use gaming in English 
teaching (Professor Zhang Lianzhong) 


6:00-6:30 p.m. 


Wrap-Up and Questions 


Wednesday, August 5 


8:30-8:45 a.m. 


Day Two Welcome 


8:45 a.m.-9:45 a.m. 


High Technology in Language Learning 

Presentation on the value of informal, media- and technology-based applications in 
the teaching and learning of language (Professor Wu Heping) 


9:45 a.m.-10:45 a.m. 


Practice Session 2 in Computer Lab 


11:00 a.m.-12:00 p.m. 


Sharing the Pilot Experience 

Presentation and discussion on how the learning object was used in class and what 
worked in engaging students in learning (teachers from pilot schools) 


12:00-2:30 p.m. 


Lunch and Break 


2:30-3:15 p.m. 


Classroom Demonstration 

Teachers who participated in the spring pilot act as model teachers while workshop 
participants act as “students” in the demonstration 


3:15-4:00 p.m. 


Discussion 

Teachers react to their experience as “students” and explore strategies for how they 
will engage and support students using the learning object 


4:15-5:30 p.m. 


Developing a Lesson 

Teachers work in small groups to develop sample lessons based on their review of 
the Teacher’s Guide, experience using The Forgotten World and classroom 
demonstrations 


5:30-6:30 p.m. 


Reporting 

Overview of teacher dashboard and student/class reporting features: how to read 
reports and strategies for using the reports to shape classroom teaching and 
individual student instruction 


6:30-8:00 p.m. 


Kick-Off Banquet 

Celebration for all teachers, workshop participants, project leaders, and local officials 


Thursday, August 6 


8:00-8:15 a.m. 


Day Three Welcome 


8:15-9:00 a.m. 


Monitoring and Evaluation Program 

Timeline from September 2009 to March/April 201 0 


9:00-9:30 a.m. 


Experiences and Lessons From Evaluation in Pilot 

Presentation and discussion of what worked during the pilot, including thoughts on 
how teachers can use the program more effectively as a supplement to their 
current teaching resources and lesson plans (Professor Zhu Yiming) 


9:45-11 :00 a.m. 


School Groups Meet to Develop Action Plan 

Headmasters and teachers collaborate in school-based small groups to discuss plans 
for working with the program in their classrooms 


11:00-11:30 a.m. 


Sharing and Review of School Action Plans 


11:30 a.m. -12:00 p.m. 


Conclusion 


12:00-12:30 p.m. 


Closing Ceremony 


Source: The U.S. -China E-Language Project. 
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Exhibit 5. Participants’ Ratings of Teacher Training Workshop 




Contentofthe Purpose of the Understanding of Level of independent Confidencein 

training program TFW TFW use teachingwith TFW 

Training elements 



Exhibit reads: Teachers were positive about the training experience but were critical of their own level of expertise in using The 
Forgotten World. 

Note: TWF = The Forgotten World 
Source: The U.S. -China E-Language Project. 

On the training evaluation forms, teachers were also given the opportunity to share 
thoughts and concerns about using The Forgotten World in their classes. Three major concerns 
were expressed. First, teachers stated that they were worried about the condition of the 
computers and the Internet connection at their schools. They also asked for continued 
professional guidance and help, including meetings with experts and continuing opportunities for 
training. Finally, several participants expressed a desire to build exchanges and connections 
outside their schools and said they hoped to have more communication with American experts. A 
few stated that they would like to learn more about American middle schools. While some of 
these activities were beyond the scope of the project, it is clear that this project could be 
extended to promote exchanges between Chinese and U.S. educators. 

Because the software and implementation team was interested in potentially sponsoring 
Web chats for the teachers, the questionnaire also includes an item asking participants how often 
they used the Internet. Nearly half of the teachers reported using the Internet every day, and 
another quarter reported using it two to three times per week, as shown in Exhibit 6. Based on 
these responses, an online forum was developed and Tencent QQ 5 was used by teachers and 
experts to exchange information. 



5 Tencent QQ is the most popular instant messaging and chat software in China. 
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Exhibit 6. Training Participants’ Self-reported Use of the Internet 



Never 




Exhibit reads: Nearly half of teachers reported using the Internet every day. 
Source: The U.S. -China E-Language Project. 



4.2 School and Student Participation 

One of the first questions to be addressed is whether the students in treatment and 
comparison schools were similar in English proficiency at the start of the study. Information on 
the number of classes and students in participating schools is presented in Exhibit 7. The first 
column shows the total number of eighth-grade classes in the school. The next four columns 
show that all students in all classes were tested in the fall and present their mean fall scores in 
reading and listening. The mean reading score of all students in treatment schools was 201.17 
compared to 200.08 in the comparison schools, while the mean listening scores were 176.08 and 
175.91, respectively. The scores for the reading scores range from 159 to 256, and the listening 
scores range from 79 to 267 [data not shown in exhibit]. The differences in mean scores between 
treatment and comparison schools were not statistically significant. 

A second question that can be answered by this table is whether students in the sample of 
classrooms that participated in the study were representative of all the students in their schools. 
Not all the classrooms in the treatment school used The Forgotten World. In two treatment 
schools, all eighth-grade English classes participated; in the remaining three schools, about half 
the eighth-grade classes participated. Teachers within the treatment schools were not randomly 
assigned to participate; rather, teachers volunteered to participate in the study. The final sample 
includes only those classrooms in treatment schools that used The Forgotten World and a random 
sample of classrooms selected from the comparison schools. 

The columns labeled “Final sample” present the numbers of classes and students 
participating in the study and their mean scores on the fall test. It appears that the performance of 
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students in classrooms participating in the study — those that used The Forgotten World — is 
similar to the student performance in all eighth-grade classrooms: the mean reading score across 
all students in treatment schools was 201.17, while the mean in the participating classrooms in 
the treatment schools was 201.31. The mean listening scores were also approximately equal. The 
differences between the total eighth grade mean scores in reading and listening and the sampled 
eighth-grade classrooms were not statistically significant. Only in Treatment School 5 does there 
appear to be the possibility that higher-performing classes were selected to participate in the 
study and use The Forgotten World, but these differences were not statistically significant. In the 
comparison schools, the total mean reading and listening scores of students in the sampled 
classrooms closely matched the mean scores of the schools. 



Exhibit 7. School and Student Participation 





Total 




All 






Final sample 


Difference 




8th- ' 
grade 
classes 






Mean fall scores 






Mean fall scores 


rvalues 




Classes 


Students 


Reading 


Listening 


Classes 


Students 


Reading 


Listening 


Reading 


Listening 


Treatment schools 
subtotal 
























School 1 


6 


6 


269 


196.69 


172.55 


6 


243 


196.96 


173.02 


.11 


.05 


School 2 


8 


8 


596 


197.84 


166.61 


8 


552 


198.13 


167.04 


.12 


.08 


School 3 


13 


13 


956 


203.11 


179.55 


6 


414 


204.51 


181.93 


.53 


.96 


School 4 


12 


12 


932 


205.14 


180.79 


6 


439 


205.64 


182.08 


.67 


.68 


School 5 


8 


8 


405 


195.32 


173.51 


4 


190 


199.22 


177.16 


1.17 


.61 


Total/mean score 


47 


47 


3,158 


201.17 


176.08 


30 


1,838 


201.31 


175.81 


.05 


-.05 


Comparison schools 
subtotal 
























School 6 


8 


8 


572 


196.64 


162.45 


6 


394 


196.69 


162.57 


.61 


.03 


School 7 


12 


12 


748 


198.56 


172.48 


7 


374 


199.08 


174.02 


.58 


.41 


School 8 


8 


8 


487 


204.76 


187.32 


6 


345 


206.42 


192.51 


.60 


.69 


School 9 


8 


8 


287 


196.64 


170.92 


4 


130 


196.05 


165.81 


-.96 


-1.52 


School 10 


12 


12 


781 


202.38 


183.73 


9 


559 


203.19 


186.06 


1.27 


.92 


Total/mean score 


48 


48 


2,875 


200.08 


175.91 


32 


1,802 


201.03 


178.21 


.40 


.31 


Difference treatment 
and comparison 
schools 








f =.45 


II 

o 

CO 






f =.10 


■'T 

CO 

II 





Exhibit reads: Treatment and comparison schools were well matched, and students in the final sample of participating classes performed about the same on fall 
assessments as all the students in their schools. The possible scores for the reading scale range from 159 to 256, and the possible scores for the listening scale range 
from 79 to 267. 

* p< .05 

Source: The U.S.-China E-Language Project. 



The characteristics of students and teachers in the treatment and comparison schools can 
be examined using data collected in the fall questionnaire. Exhibit 8 presents information on key 
student demographics and on the students’ experience using English. The middle schools studied 
had more male students (55 percent) than female students (45 percent) , and most were born in 
1995 or 1996, which makes them slightly older than typical eighth-grade students in the United 
States. 



Most students (nearly 70 percent) in the treatment group reported that they began 
learning English in third grade. Typically, students are introduced to English in elementary 
school; the frequency and intensity of instruction increase when students begin middle school in 
seventh grade, and the majority of students, more than 80 percent, expected to attend college. 
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Approximately half of the students reported having computers at home, having Internet access, 
and playing computer games once a week or more. 



Exhibit 8. Characteristics of Students 



The Forgotten World 

schools Comparison schools 



Percentage of students who report: 


Percent 


Number 


Percent 


Number 


Difference 


tvalue 


Sex 














Female 


44.61 


3,080 


47.03 


2,815 


-2.42 


-1.23 


Year of birth 




3,040 




2,795 






Before 1995 


10.07 




9.23 








1995 


51.41 




52.77 




-1.36 


-.45 


1996 


36.94 




36.92 




0.02 


.00 


After 1 996 


1.58 




1.08 








Grade began learning English 




3,029 




2,792 






Before third grade 


4.72 




16.12 








Third grade 


68.67 




59.46 




9.21 


1.22 


Fourth through seventh grade 


14.16 




13.47 








Eighth grade 


12.45 




10.96 




1.49 


1.12 


Expecting to attend university 


83.71 


3,063 


80.53 


2,815 


3.18 


1.02 


Flaving computer at home 


50.65 


3,068 


42.93 


2,816 


6.42 


-.41 


Flaving Internet access at home 


43.11 


3,055 


49.22 


2,812 


-6.11 


-.45 


Playing computer games once a week or more 


58.15 


3,068 


54.38 


2,803 


3.81 


.71 


They have ever been tutored in English 


90.76 


3,075 


88.36 


2,819 


2.4 


.70 


Reading English-language books, magazines, 

newspapers, and websites once a week or more 


70.54 


2,872 


66.69 


2,594 


3.85 


1.46 


Using English to speak with friends/family 


43.9 


2,861 


48.42 


2,594 


-4.52* 


-2.12 


Playing computer games with English once a week 
or more 


41.33 


3,075 


39.52 


2,816 


1.81 


.40 


Watching English TV once a week or more 


40.11 


3,069 


37.13 


2,817 


2.98 


1.02 


Chatting online in English once a week or more 


15.34 


3,070 


13.55 


2,820 


1.79 


.77 


They have traveled to English-speaking country 


3.45 


2,872 


5.74 


2,594 


2.29 


-1.21 



Exhibit reads: Characteristics of students were similar in treatment and comparison schools. 
*p< .05 

Source: The U.S. -China E-Language Project. 



Several questions were asked of students to determine their prior experience in learning 
and using English outside of school. About 90 percent of students reported that they had received 
tutoring in English at some time, and 70 percent reported that they read books, magazines, 
newspapers, or websites in English. Approximately 40 percent sometimes use English to speak 
with family or friends, and a similar percentage play computer games with English or watch 
English-language television programs at least once a week. Students in treatment schools were 
significantly less likely than students in comparison schools to speak English with family and 
friends. Only a small number, 3 percent, reported having traveled to an English-speaking 
country. 

English teachers in this study were predominantly female and have an average of 
13 years’ experience teaching English, as shown in Exhibit 9. The majority (about 79 percent) 
have four-year degrees in English. The average age is mid-thirties. Significantly more teachers in 
the treatment schools were female than in the comparison schools 
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Exhibit 9. Characteristics of Teachers 



The Forgotten World 

schools Comparison schools 



Characteristic 


Percent 


Number 


Percent 


Number 


Difference 


t value 


Percent of teachers whose highest degree 
is four-year degree or above in English 


78.95 


19 


86.36 


22 


7.42 


t=-1 .04 


Mean age of teachers 


36.38 


24 


33.58 


24 


2.79 


t=1 .47 


Percent female 


91.67 


24 


70.83 


24 


20.83* 


t=2.6 


Mean years teaching English 


13.33 


24 


10.46 


24 


2.87 


t=1 .19 



Exhibit reads: Characteristics of teachers were similar in treatment and comparison schools. 
*p< .05 

Source: The U.S. -China E-Language Project. 



4.3 Implementation of The Forgotten World 

Early in the fall, it became apparent from teacher logs and e-mails to support staff that the 
Internet connections in the schools were insufficient to support large groups of students using 
The Forgotten World simultaneously. While the software development team had visited all the 
computer labs in August 2009, when school was not in session, this problem surfaced in late 
September 2009, when the system was used simultaneously by large numbers of students. 
Students would log in to the application but would often lose their connection and be forced to 
log in a second time. Personnel in the province worked to diagnose the issue, but the problem 
could not be fixed without the purchase of new servers for the schools. The budget for 
implementation provided by the William and Flora Hewlett Foundation did not include the 
purchase of equipment. In December 2009, the Dell Corporation agreed to donate servers to the 
schools. During fall 2009, some teachers had continued to use The Forgotten World, but system 
performance was frustrating to teachers and students, as documented by site visits and teaching 
logs. 

Chinese schools typically have a long holiday to celebrate the New Year. In this 
province, schools closed January 18, 2010, and reopened March 1, 2010. Thus, the new servers 
were not used until March 2010. Because of the initial difficulties and the long break, the project 
team decided that the project needed a jump start, and the lead U.S. software developer traveled 
to China to visit schools and conduct supplementary training in early March 2010. Reports from 
the training were positive, and a site visit by ECNU confirmed that teachers were enthusiastic 
about using the application now that the Internet connections were stable and response times had 
improved. 

Information on the frequency of use of The Forgotten World in participating classes was 
collected from multiple sources, as presented in Exhibit 10. Two sources of information are 
presented in this table: student reports of how frequently they used The Forgotten World both in 
and out of class and teacher reports of how frequently they used The Forgotten World in class 
and how they organized students to use it outside of class. 
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Exhibit 10. Reported Use of The Forgotten World by Students and Teachers 



Percentage of students and teachers who used 
The Forgotten World once a week or more 



The Forgotten l/l/or/c/schools subtotal 


In class 
(reported by 
students) 


Outside of class 
(reported by 
students) 


In class 
(reported by 
teachers) 


Outside of class 
(reported by 
teachers) 


School 1 


78.92 


17.30 


100.00 


33.33 


School 2 


70.32 


12.12 


50.00 


0.00 


School 3 


71.96 


24.32 


80.00 


40.00 


School 4 


84.24 


20.52 


100.00 


50.00 


School 5 


84.95 


27.03 


75.00 


25.00 


Total n 


1,728.00 


1,725.00 


22.00 


22.00 



Exhibit reads: Over 70 percent of students reported that they used The Forgotten World at least once a week in class. 
Source: The U.S. -China E-Language Project. 



Schools participating in the study were asked to provide students with the opportunity to 
use The Forgotten World at least once each week in class. Over 70 percent of students reported 
that they used The Forgotten World once a week or more in schools, and about 20 percent 
reported that they used it once a week or more outside of class. About 81 percent of teachers 
reported that they used The Forgotten World once a week or more in class, and about 29 percent 
of teachers reported they asked students to use the program outside of class. It is not surprising 
that student reports of use are somewhat lower than teachers’ reports, because the size of the labs 
required students to share computers at times, and because students may not have participated in 
each session. 

In June 2010, teachers from each school reported on their progress in using The 
Forgotten World. One school had completed all 12 episodes, all schools had completed at least 7 
episodes, and most had completed 8-10 episodes. 6 

Clearly, the implementation of The Forgotten World was incomplete. Because of 
technological limitations, use during the first semester was limited. However, use during the 
second semester approached project goals, with nearly three-quarters of all students reporting 
that they used the program at least once a week. 

4.4 Teachers’ Classroom Use of The Forgotten World 

The teacher logs provide information on how The Forgotten World was used in classes 
from March through June 2010. Typically, teachers reported using The Forgotten World once a 
week with classes, for approximately 40-45 minutes. Time spent on completing one episode 
varied across schools and teachers: the reported range was from one to six class periods to 
complete one episode of The Forgotten World. 

Before class, teachers reported spending time to get familiar with the stories and games 
and design questions and activities to help students learn the story, the language points, the 
sentence structures, and practice conversations. Teachers would also check network connections 



6 Teachers in treatment school 5 had finished teaching all 12 episodes. Treatment school 1 reached 
Episode 10; classes at treatment school 4 had reached Episodes 8-10; treatment school 3 was at Episodes 10-11; 
and treatment school 2 was at Episode 7 in Class 7; Episode 9 in Classes 1, 2, 3, and 8; and Episode 12 in Classes 4, 
5, and 6. 
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and website log-in conditions before class. Students were reminded before the start of class that 
their user names should not be shared with other people. 

In class, teachers typically began by demonstrating the story on the teacher’s computer 
projected to a large screen, while explaining the stories to the students, pointing out specific 
knowledge points and sentence structures, and teaching students how to play the games. After 
the teacher’s demonstration, students would learn the stories, mimic and practice conversations 
on their own, and play the games on their individual computers. At the end of class, teachers 
sometimes would assign homework to the students. 

The teacher logs indicate that students became enthusiastic about the program after the 
network condition improved. Students were interested in the stories and games, and in carrying 
out conversations. Teachers reported that some students found games more interesting than 
stories. The majority of students were involved in the class, but they progressed at different rates, 
making classroom management difficult. Teachers sometimes asked students who went faster to 
form small discussion groups so students learning at a slower pace could have time to finish the 
story. 
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5.1 Student Results Based on the Assessments and Questionnaires 

Did students in classrooms that used The Forgotten World improve their English reading 
and listening skills more than students in classrooms in comparison schools that did not have 
access to the program? Overall, students participating in the treatment schools did not increase 
their proficiency more than students in the comparison schools. However, it appears that students 
with relatively low levels of proficiency (those who scored below the median on the fall 
proficiency tests) made greater gains in the treatment schools than in the comparison schools. 
Exhibit 1 1 presents information on the student scores at the beginning and end of the year on the 
reading and listening assessments. This table includes all students who were in the final sample, 
regardless of their reported use of The Forgotten World. The mean gain in reading for lower- 
performing students was 8.5 in treatment schools compared with 5.7 in comparison schools. In 
listening, the lower-performing students gained 9 points in treatment schools and 5.5 points in 
the comparison schools. These differences were statistically significant. This finding was 
confirmed by use of a hierarchical linear model, presented in Appendix H. 



Exhibit 11. Mean Performance on Assessments 

The Forgotten World schools Comparison schools 

Mean Mean 

individual individual 





Beginning 
of year 


End of 
year 


gain or 
loss 


Beginning 
of year 


End of 
year 


gain or 
loss 


Difference 


t value 


Fall reading score 


















Bottom half (below the 
median) 


194.45 


203.05 


8.48 


194.46 


200.23 


5.70 


2.78* 


2.28 


Top half (above the 
median) 


207.41 


214.65 


7.44 


207.19 


215.37 


7.96 


.52 


-.04 


Reading mean 


201.31 


209.28 


7.92 


201.03 


207.85 


6.85 


1.07 


.88 


Total n 






1,810 






1,782 






Fall listening score 


















Bottom half (below the 
median) 


164.39 


173.53 


9.03 


164.50 


170.09 


5.47 


3.56* 


2.75 


Top half (above the 
median) 


185.61 


188.99 


3.35 


191.59 


195.80 


4.15 


-.8 


-.51 


Listening mean 


175.81 


181.79 


5.97 


178.21 


183.01 


4.80 


1.17 


.87 



Total n 1,824 1,786 

Exhibit reads: Students who began the year in the bottom half of the sample increased their English proficiency more in treatment 
schools than in comparison schools. 

*p< .05 

Source: The U.S. -China E-Language Project. 



Did students who reported using The Forgotten World more often exhibit increased 
English language proficiency? Exhibit 12 examines student performance on beginning and end- 
of-year assessments by how frequently they reported using The Forgotten World. The table 
suggests that the students who used The Forgotten World more frequently may have made 
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slightly more gains in reading and listening proficiency than students who never used the 
program or used it only once or twice. While the results are in the expected direction, with this 
sample size, the results are not statistically significant. 



Exhibit 12. Performance on Assessments, by Use of The Forgotten World 

The Forgotten l/l/or/a'schools 



Mean gain or loss 



Used The Forgotten World . . . 


Beginning of year 


End of year 




Never (n = 82) 


Reading score 


197.59 


201.47 


3.89 


Listening score 


172.63 


172.56 


-0.07 


Once every few months (n = 101) 


Reading score 


197.23 


202.93 


5.74 


Listening score 


171.92 


174.86 


3.16 


Once or twice a month (n = 221 ) 


Reading score 


198.99 


206.69 


7.68 


Listening score 


172.26 


176.75 


4.49 


Once or twice a week (n = 1 283) 


Reading score 


202.69 


211.22 


8.53 


Listening score 


177.63 


184.61 


6.96 


Three to four times a week ( n = 32) 


Reading score 


202. .09 


209.25 


7.15 


Listening score 


173.06 


184.21 


11.16 


Every day ( n = 9) 


Reading score 


199.66 


206.33 


6.67 


Listening score 


168.33 


173.89 


5.56 



Exhibit reads: Relationship between frequency of use and improved performance on assessment is not statistically 
significant. 

Source: The U.S. -China E-Language Project. 



Did using The Forgotten World have an effect on students 9 interest in or motivation to 
learn English? Language experts and teachers in China repeatedly told the study team that 
language learning is not popular in China, even though it is important for success in school. 
Research on teaching English as a foreign language has identified motivation as an important 
factor (see, e.g., Norris-Holt 2002), and there is a long tradition of research that links student 
motivation and academic achievement (Dalton 2010). Thus, one of the goals of this study was to 
examine whether this approach to language learning might have an effect on student interest in 
and motivation to learn English. 

Almost all middle school students, approximately 95 percent, who participated in this 
study believed that studying English is somewhat or very important, as shown in Exhibit 13. 
However, fewer students expressed interest in learning English, and it appears that interest 
waned for both groups as the year progressed. 
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Exhibit 13. Importance of and Interest in Learning English 



The Forgotten World schools Comparison schools 



Beginning of year End of year Beginning of year End of year 





Percent 


Number 


Percent 


Number 


Percent 


Number 


Percent 


Number 


Importance of learning English 
Interest in learning English 


96.44 

70.47 


1,798 

1,798 


92.87 

60.90 


1,794 

1,793 


94.60 

67.60 


1,778 

1,778 


90.19 

54.53 


1,784 

1,779 



Exhibit reads: Nearly all students reported that learning English is somewhat or very important. 
Source: The U.S. -China E-Language Project. 



At the end of the year, students were asked about their motivation to learn English, and 
how much fun it was to leam English. Fifty-one percent of students in the treatment schools said 
that motivation to learn English had improved, compared to 40 percent of students in comparison 
schools, as shown in Exhibit 14. From the outset of the study, some Chinese experts suggested 
that increasing student motivation to learn English might be one of the most important benefits 
of using The Forgotten World. Given the relatively limited exposure that students had to the 
program, the suggestion that this might be the case is encouraging. 



Exhibit 14. Percentage of Students Who Reported That Motivation to Learn English Improved 
and That Learning English Is Fun 




■ TFW Schools 

■ Comparison schools 



Motivation to learn Learning English is fun 
English improved 



Exhibit reads: A higher percentage of students in schools that used The Forgotten World reported that motivation improved 
compared to students in comparison schools. 

Source: The U.S. -China E-Language Project. 

Note: Percentage motivated (.5092; SE = .01 753) in treatment schools is higher than percentage motivated (.3978: SE = .0406) in 
comparison schools; t = 2.52 and p < .05. Difference in fun between groups is not significant. 



What were student attitudes toward The Forgotten World? About 40 percent of 
students reported that they never used The Forgotten World outside of school, and about the 
same number reported that they used the program outside of school once a week or more, as 
shown in Exhibit 15. Students also provided mixed reports on the helpfulness of the program: 
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46 percent said that it was somewhat or very helpful, 42 percent said it was not helpful, and 
12 percent said they didn’t know or hadn’t used it much. 

Students reported that they faced a number of challenges in using The Forgotten World. 
The major challenges, reported by over half the students, involved not having enough time to 
engage in the episodes (64 percent) and problems with the computers at school (57 percent). 
Concerns about the possible negative effect of The Forgotten World on test scores were voiced 
by a small number of students, about 10 percent. 



Exhibit 15. Student Experience With The Forgotten World 





Percent 


Number 


Use of The Forgotten World outside of school (Q14) 




1,722 


Never 


40.29 




Twice a month or less 


19.19 




Once a week or more 


40.52 




How helpful was The Forgotten World ? 




1,723 


Don’t know or didn’t use much 


12.19 




Not helpful 


41.96 




Somewhat helpful 


37.55 




Very helpful 


8.30 




What was biggest challenge? (multiple answers allowed) 






Insufficient time for using the program 


64.60 


1,723 


Limited support from school 


23.51 


1,723 


Problems with computers at the school computer lab 


57.05 


1,723 


Concerns with negative impact on test scores 


9.81 


1,723 


Other challenges 


14.14 


1,722 


Exhibit reads: Students reported mixed opinions on whether The Forgotten World was helpful. 
Source: The U.S. -China E-Language Project. 



Students were given the opportunity to specify other challenges they faced in using the 
program. The challenges can be categorized into four groups, related to (1) the program content 
and structure, (2) lack of access to the software, (3) hardware and connectivity issues, and 
(4) personal issues. Challenges mentioned by more than 10 students each included the following: 

Program content and structure issues 

• Difficulty understanding sentences 

• Too many unknown vocabulary words 

• Insufficient Chinese translation 

• Speech too fast to understand 

• Dictionary slow to use, not always helpful 
Lack of access issues 

• Teachers did not always hold class 

• Insufficient computers for all students in class 

• Not enough time to use application 

• No computer at home 
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Hardware and connectivity issues 

• Internet speed was slow. 

• Computers sometimes were automatically logged out. 

• Computers sometimes froze. 

• Computers sometimes failed to log in. 

5.2 Teacher Attitudes and Experiences 

What were teacher attitudes toward The Forgotten World? At the start of the project, 
the TWG and team discussed how The Forgotten World should be delivered to students. One 
group felt that it was best as a standalone application that could be accessed directly by students; 
the other group felt that teachers’ use of the program was essential. Both groups were concerned 
about teachers’ receptivity to the new approach but thought that implementing this approach in 
classrooms would afford teachers the opportunity to think about their teaching in novel ways. 

Teachers’ opinions on the helpfulness of The Forgotten World were mixed, as were 
student opinions. As presented in Exhibit 16, some 59 percent reported that it was somewhat 
helpful, and 41 percent reported that it was not very helpful. No teachers chose the more extreme 
options of “very helpful” or “not at all helpful.” 

A majority of teachers said they were unlikely to use The Forgotten World, or any other 
technology approach, in the future: only 14 percent said they were likely to use The Forgotten 
World again, and only 18 percent reported that they were likely to use some other software. 
(Reasons teachers gave for these responses are presented in Exhibit 17.) 

Teachers reported noticing a change in students’ motivation to learn English, with only 
5 percent reporting that they noticed no change. This is consistent with the finding that students 
in treatment schools reported higher levels of motivation than students in comparison schools. In 
light of some of the comments from students about the difficulty of the program, it is interesting 
that more than 70 percent of teachers said that they would recommend that The Forgotten World 
be used for eighth-grade students, rather than seventh- or ninth-grade students. 

Teachers were asked follow-up questions to two of the items reported in Exhibit 16: to 
explain why they were likely or not likely to use The Forgotten World in the future, and to 
provide suggestions about how The Forgotten World might be improved. Exhibit 17 presents 
some of the reasons that teachers are likely or unlikely to use the program in the future. Among 
those who said they were not too likely to use it again, the most common reason cited was that it 
represented too much additional content and vocabulary to cover, given the time available. A few 
also mentioned that preparing students for entrance exams was a priority. Among those teachers 
who said they were likely to use the program again, the major reason cited was that it generated 
interest and enthusiasm on the part of students. 
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Exhibit 16. Teacher Experience With The Forgotten World 


Percent 


How helpful was The Forgotten World ? (n = 22) 




Not helpful at all 


0.00 


Not very helpful 


40.91 


Somewhat helpful 


59.09 


Very helpful 


0.00 


How likely to use The Forgotten World in future (n = 22) 




Not likely at all 


0.00 


Not very likely 


59.09 


Somewhat likely 


27.27 


Very likely 


13.64 


How likely to use other software (n = 22) 




Not likely at all 


0.00 


Not very likely 


40.91 


Somewhat likely 


40.91 


Very likely 


18.18 


Has student motivation changed since using The Forgotten World ? (n = 22) 




No change 


4.55 


A little more motivated 


81.82 


Somewhat more motivated 


13.64 


A lot more motivated 


0.00 


For what grade level would you recommend The Forgotten World ? ( n = 22) 




Seventh grade 


19.05 


Eighth grade 


71.43 


Ninth grade 


9.52 


Exhibit reads: Teacher reaction to helpfulness of The Forgotten World was mixed, but they did report noticing that students were 


more motivated. 




Source: The U.S. -China E-Language Project. 





Specific suggestions for improving The Forgotten World are presented in Exhibit 18. It is 
clear from the comments that the quality of the schools’ computer laboratories and the 
connectivity to the Internet were major issues for the teachers. The size of the laboratories, 
relative to the number of students in class, was also problematic. Perhaps because the program 
was only implemented for a single semester, teachers reported that students did not have enough 
time to use the program. There also appear to be a number of areas in which the software 
application might be improved, including allowing teachers and students to adjust the speed of 
the speech contained in the program, provide clear and simple instructions for the games, and 
provide more support for the vocabulary presented in the program. A few teachers also 
commented on the quality of the story and episodes. 
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Exhibit 17. Teacher Comments About Why They Are Likely or Not Likely to Use The 
Forgotten World in the Future 



How likely are you to use The Forgotten World in the future, and why? 



Not likely 



Comment 1 
Not too likely 

Comment 1 
Comment 2 

Comment 3 

Comment 4 

Comment 5 

Comment 6 

Comment 7 

Comment 8 

Comment 9 

Comment 10 
Comment 1 1 



None 



Some network issues impacted the teaching activities. 

The games and content of the software are rarely related with the test. School and 
parents have the requirement for students to do well on the test scores. 

The game design could not attract student's attention when using; there was a difficulty 
in organizing the class. 

The Forgotten World does not have a systematic knowledge structure, so it cannot be 
part of the curriculum. 

There was a lot of content and not enough class hours, The Forgotten World could only 
be used as a supplementary material. 

Teaching tasks were too much. There were not enough class hours. Work load was too 
much. There were too many students in each class. 

1 . There is already a lot of content in student textbooks; 2. Majority of the contents in 
this software are in reading. 

The textbooks which we are currently using are comprehensively designed, plus there 
were not enough class hours for The Forgotten World, and it could only be used as 
a supplementary materials. 

Too much new vocabulary, colloquial, not suitable for exam preparation. 

Factor of high school entrance exam. 

Since there are many limitations, The Forgotten World is very likely to be used only as 
supplementary curriculum. 



Somewhat likely 
Comment 1 

Comment 2 

Comment 3 

Comment 4 
Comment 5 



It attracts students by combining computer online operation with English; Stories are 
intricate and abundant, which fit the age group of the students. 

It significantly increased the enthusiasm of students in learning English; it could 
increase the comprehensive quality of students in full scale. 

To great extent The Forgotten World helps the students develop and increase their 
interest in learning English. Meanwhile, it is a very effective supplementary means 
of teaching. 

The Forgotten World is a beneficial supplement of current English-language teaching. 

English teaching should fuse language teaching into the context of language. 



Very likely 

Comment 1 
Comment 2 



Students are very interested in it. It is an inevitable trend to improve out-dated 
hardware. 

The program is a combination of information technology and test preparation. 



Exhibit reads: Teachers’ reasons for using or not using The Forgotten World \r\ the future. 
Source: The U.S. -China E-Language Project. 
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Exhibit 18. Teacher Suggestions About Improving The Forgotten World 



Comment 1 



Comment 2 
Comment 3 

Comment 4 

Comment 5 
Comment 6 

Comment 7 



Comment 8 
Comment 9 
Comment 10 

Comment 1 1 
Comment 12 

Comment 13 
Comment 14 
Comment 15 
Comment 16 

Comment 17 



1 . School must provide appropriate hardware so as to guarantee every class goes smoothly. 
Otherwise, it will be like the current situation where a majority of students had to log in 
repetitively, which discouraged students; 2. In the software, the pronunciation, tone, and 
speed of some sentences are very difficult; 3. Teachers must communicate and cooperate; 
Avoid individual and separate work; 4. Teachers need to make efforts to familiarize the 
texts; 5. Students must have sufficient time to use computer. 

Improve outdated teaching equipment; obtain support from parents; increase opportunity and 
time for students to practice speaking on certain subjects out of class. 

1 . Excited students’ learning interest by explaining the stories; 2. Help students learn new 
vocabulary and phrases by playing games; 3. It would be better if the hardware at school 
was improved. 

1 . Increase time to practice and learn; 2. Design The Forgotten World to be actual listening and 
speaking curricula so as to excite students so that they will take it more seriously; 3. 

Provide more training and learning opportunities to teachers in order to increase teaching 
level and professional quality. 

1 . Ensure the network runs smoothly; 2. Make stories more exciting so students will be more 
interested in it; 3. Make the stories shorter; design quiz after each episode; 4. The games 
should be simpler so students can enjoy success. 

1 . Some episodes/stories are too long. It will be better to divide it into several shorter episodes; 
2. Decrease the amount of vocabulary in games such as TRIGANGLE so that majority of 
the students could enjoy each class and increase their interest and confidence learning 
English; 3. It will be better if the stories are closer to real life and practice. 

The software design should stay closer to the textbook to be its extension, so that students can 
grasp the textbook contents when playing the games. The software should start from lower 
level. It’s hard to correct eighth-grade students’ pronunciation; the conversation speed in 
the games is pretty fast and students have limited time to use computer, so it's hard for 
students to improve. 

The episodes and games should match with the textbooks, which would be beneficial to 
improving students’ test scores and the teachers’ rate of using the software. 

1 . The instructions for games should be more clear and specific; 2. If episodes were designed 
with a cartoon effect, it would be easier. 

1 . Need to have collation of English and Chinese. There should be some Chinese notes for the 
difficult sentences; 2. Listening speed should be adjustable from slow to fast; 3. Correct 
instructions should be provided for the games. 

There should be simple textbooks to help students in learning English designed to help 
students learn English while playing. Need to help students use computer with purpose. 

1 . The contents should stay close with actual lives and current time; 2. There should be a 
system of grammar; 3. The difficulty level should go from easy to hard gradually; 4. The 
story should be vivid and interesting; 5. It will be better to design it as a cartoon; 6. Games 
should combine knowledge and ease of playing to help student learn. 

1 . Need to make sure there is more time to use computer; 2. Enforce supervision and 
management when students use computers. 

Provide students enough time and provide teachers enough freedom and time to look into the 
contents. 

Students must be self-disciplined and willing to participate; there should be enough time every 
time to finish the task. 

Must make sure every student has a computer; There should be at least 3-4 classes every 
week; the electronic dictionary should provide entries that cover all the new vocabulary 
words students will encounter when learning the stories. 

It is better to design the game in animation, make the stories more interesting to increase 
student interest. 



Comment 18 There should be computers available in the classrooms which can be used at any time. 

See notes at end of table. Cont’d next page 
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Exhibit 18. Teacher Suggestions About Improving The Forgotten World — Continued 



Comment 19 There is a conflict between high school entrance exam and language application; the sections 

on grammar and vocabulary in the high school exams are very detailed, and the proportion 
of the score based on listening and oral communication is relatively small. The Forgotten 
World provides an opportunity for the students to practice listening skills and develop 
language sense. 

Comment 20 The interface should be more user friendly. The games should be more interesting. Need to 

make the software run more smoothly. The problems encountered during usage should be 
addressed in a timely manner. 

Exhibit reads: Teachers provided suggestions about improving The Forgotten World. 

Source: The U.S. -China E-Language Project. 



One of the hopes of the Chinese experts who advised the project team was that The 
Forgotten World would provide with teachers an opportunity to collaborate and to rethink their 
approach to teaching. As shown in Exhibit 19, some 77 percent of teachers who participated in 
this project reported collaborating with other teachers at least once a week. Even more striking is 
the fact that 95 percent of participating teachers reported that it changed the way they think about 
teaching. Note that the fact that teachers reported that it changed the way they think about 
teaching may be the result of participating in a new project, and may not necessarily be due to 
the particular aspects of this approach. 



Exhibit 19. Teacher Experience With The Forgotten World 



Percent 

How often did you collaborate with other teachers while using The Forgotten World ? (n= 22) 



Never or once every few months 0.00 

Once or twice a month 22.73 

Once or twice a week 54.55 

Three times a week or more 22.73 

Has using The Forgotten World changed the way you think about teaching? (n=22) 

Yes 95.45 

No 4.55 



Exhibit reads: Nearly all teachers reported that using The Forgotten World changed the way they think about teaching. 

Source: The U.S. -China E-Language Project. 

Teachers were asked a follow-up question to explain how using this approach changed 
the way they think about teaching. Exhibit 20 presents their answers. Many teachers referred to 
the affective impacts of the approach, using words and phrases like “spontaneous,” “happy 
learning experience,” “class became dynamic,” “encourages students,” “apply games and 
entertainment in teaching,” “nurture interest,” and “feel native English.” A secondary thread 
through the teacher comments emerges in the references to students’ active role in learning; 
“reciprocity between teachers and students,” “they learn from experience and application,” “we 
should return the classroom to the student.” 
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Exhibit 20. Teachers’ Comments on How This Approach to Language Changed the Way 
They Think About Teaching 



If The Forgotten World has changed the way you think about teaching, please explain 



Comment 1 

Comment 2 
Comment 3 

Comment 4 



Innovation of teaching concepts. The Forgotten World combines and connects 
with current task-oriented teaching method to reflect the teaching process 
where teachers take the lead, students act as major component, and there is 
reciprocity between teachers and students. 

In English learning, students attempt to transition to learning spontaneously in 
specific language contexts. They learn from experience and application. 

The Forgotten World at least promotes us to change perceptions away from 

traditional teaching models; it’s applicable to current education reform system; 
it provides students a new experience. 

The Forgotten World provides me new teaching concepts; by letting students 
have a happy learning experience, the whole class became dynamic based 
on this new teaching concept. 



Comment 5 
Comment 6 
Comment 7 
Comment 8 

Comment 9 

Comment 10 
Comment 1 1 
Comment 12 
Comment 13 
Comment 14 
Comment 15 
Comment 16 



It encourages students to learn English for applying it. 

We should return the classroom to the students to let them speak more. 
Strengthen listening and speaking training. 

Infiltrate new teaching concepts to teaching practice; apply games and 
entertainment in teaching. 

Audio-visual characteristics in multimedia excited students’ interest in learning 
English. 

Improve ability in listening and speaking by imitating native English. 

If there is no knowledge atmosphere, there is no language. 

Should emphasize speaking and listening practice. 

Electronic textbooks attract students. 

Suggest promoting this software in developed cities. 

Nurture interest in remembering vocabulary, listening, and language sense. 

Do not only focus on exams, but let students feel the native English. 



Source: The U.S. -China E-Language Project. 
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6.1 Summary of Findings 

Several implementation issues encountered during the study significantly reduced the 
amount of time students could spend on The Forgotten World. The expected frequency and 
intensity of implementation was not realized until halfway through the study, after information 
technology infrastructure improvements were made at treatment schools. Even then, only about 
70 percent of students reported that they used The Forgotten World at least once a week. This 
partial implementation during one semester represents a fairly limited treatment in comparison 
with other education studies of this type. 

While overall, students in treatment and comparison schools performed similarly on the 
proficiency tests administered at the beginning and end of the school year, students in treatment 
schools with relatively low levels of initial proficiency (those who scored in the bottom half of 
all students on the fall tests) evidenced larger gains in reading and listening than did similar 
students in comparison schools. This finding was statistically significant for both reading and 
listening. 

Students in the treatment group reported higher levels of motivation to learn English than 
students in the comparison schools. At the end of the year, 51 percent of students in the treatment 
schools reported that their motivation to learn English had improved, compared with 40 percent 
of students in the comparison schools. Teachers in treatment schools also reported that students’ 
motivation improved. This is an important finding because motivation to learn a foreign 
language has been identified by Chinese language experts as a problem in Chinese schools and a 
barrier to learning. 

Almost all of the teachers in the treatment schools (95 percent) who participated in the 
project reported that their use of The Forgotten World changed the way they think about 
teaching. The approach to language teaching and learning that forms the basis for The Forgotten 
World is learner-centered and interactive, and it presents students with problem-based tasks and 
immediate feedback. The teachers appreciated the fact that the program provided students with 
fun opportunities for practicing listening and speaking skills, as well as exposure to native 
English speakers. However some teachers expressed concern that their primary responsibility 
was to prepare students for the English portion of the high-school entrance exam that students 
take in ninth grade, and they did not feel that use of The Forgotten World would improve student 
performance on the exam. 

6.2 Implications for the Future 

Because of the difficulty with implementation, it is difficult to judge the true efficacy of 
The Forgotten World. However, the project team was able to gather information on the issues 
that should be addressed before distributing the program more broadly in China, or elsewhere. 
The issues to be addressed fall into three major categories: hardware, teacher training and 
support, and program issues. 

Hardware issues. Many times in studies such as this, the implementation of a program 
fails because teachers or administrators are not committed to the project. This was not the case in 
this study; teachers and administrators showed remarkable perseverance in using The Forgotten 
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World, even in the face of major challenges. Two of these — the quality of the computer labs and 
Internet connections, and the size of computer labs relative to average class size — must be 
addressed to make the implementation of this or any other technology-based approach to 
language learning feasible. Neither of these issues has a cheap or easy fix. Thus, for the 
immediate future, only a subset of schools in China can be expected to implement the program 
without a major investment in upgrades to their computer labs. To complicate this matter, in 
many eastern Chinese cities, which have stronger computing infrastructures, English instruction 
begins at an earlier age, and the content of The Forgotten World would be too easy for middle 
school students in those cities. Identifying schools that have the right constellation of conditions 
for implementation is an important but challenging task. 

Teacher training and support. Implementing a new approach to teaching and learning 
requires training, support, and time. While teachers were satisfied with the initial training, from 
their logs and answers to questionnaire items it is clear that several areas of training and support 
could be improved. The software development and implementation team prepared a 
comprehensive teachers’ guide that provided teachers with sample lesson plans as well as other 
practical information, such as the vocabulary used in each episode. However, teachers still felt 
uncomfortable with some aspects of using The Forgotten World in their classrooms. 

First, teachers noted in logs, site visits, and questionnaires that classroom management in 
the computer lab was difficult. Because of the large class sizes observed in these middle schools, 
up to 80 students in a single class, managing independent activity by students may be more 
challenging than it would be with a smaller number of students. Students had many questions for 
teachers, and the teachers sometimes felt they could not answer all questions. Also, teachers 
reported that the program was too difficult for some students and that supervising those students 
added to the difficulty in controlling the class. Training specifically designed to help teachers 
work successfully in such environments might be beneficial. 

Another related area for additional teacher workshops may be the area of differentiated 
instruction — that is, tailoring instructional content to a diverse group of learners within a single 
classroom. In the United States, a considerable body of literature discusses strategies for teachers 
to target instruction to learners at different levels within the same classroom. There may be 
lessons that can be transferred from the research literature on this topic and applied to this issue. 
In the classes described by many teachers in this study, whole-group instruction was sometimes 
favored over more individualized work with small groups; again, this may be due in part to the 
large class size. Finally, in keeping with the teachers’ own judgments of the initial training, more 
time should be spent on providing teachers with hands-on experience using the software. 

Program issues. Given the issues with Internet connectivity, it is difficult to evaluate the 
strengths and weaknesses of the program. However, several issues have been identified 
repeatedly by students and teachers as impediments to students’ progress through the episodes. 

The first issue involves the talking speed of the characters speaking English in the stories 
and games. Teachers and students in the pilot study and full-scale study reported that it was 
sometimes difficult to understand stories and games because characters spoke too quickly to be 
understood. When asked about this after the pilot, the software development team reported that 
U.S. advisors had suggested that the characters speak at a speed that is typical for native English 
speakers. While this may be the ultimate goal, it seems that it would be prudent to incorporate a 
developmental approach to meeting this goal and allow students to initially select slower speeds 
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and then increase the speed as they progress. At a minimum, it might be beneficial to reduce the 
speed of speech presented in the first few episodes. 

Because of the number of new vocabulary words introduced to students in The Forgotten 
World, students and teachers expressed concern about the functionality of the dictionary that is 
available to students within the application. Some students expressed concern that it was 
inefficient and time-consuming to look up each word of a sentence they did not understand and 
suggested that students should be able to access complete translations of each sentence. While 
teachers thought that the eighth grade was the correct target grade for the program, many stated 
that eighth-grade students required support in understanding the new vocabulary and suggested 
that more efficient access to translations of new vocabulary or phrases be provided. 

The extent to which translations should be easily accessible is a source of some debate. 
Experts consulted by the software development team suggested that translations should be kept 
to a minimum so that students would not be able to rely on Chinese to work their way through 
the stories and games. Given the reaction of students and teachers, it might be advisable to adopt 
a developmental approach again and to provide complete translations of the initial few episodes 
so that students understand the basic story line. However, given that it appears that the lower- 
performing students made the largest gains, this may not be necessary. 

Finally, the extent to which students and teachers could adequately track progress 
through the program is unclear. One of the advantages of computer applications such as this is 
the ability to provide feedback on student progress to both students and their teachers. Because 
of the difficulty in U.S. servers’ capturing information on usage, it is possible that feedback to 
students and teachers did not adequately portray students’ progress through the program. This 
problem is relatively easy to fix. Capturing this information on local servers is probably a better 
option if and when large-scale adoption of The Forgotten World is considered and evaluation 
data are no longer necessary. 

The statistically significant positive results of using The Forgotten World for the lower- 
performing students along with the positive effects on student motivation is encouraging for an 
intervention that was only partially implemented. Further research could be conducted to help 
understand why the program produced these benefits, how to overcome implementation 
problems in schools lacking sophisticated computer technology, and why The Forgotten World 
did not produce positive benefits for the initially higher performers. 
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Appendix A: 

Open Language Learning Initiative Evaluation Planning Group Meeting Summary 

Beijing, China 
May 10-15, 2007 

Participants: 

U.S.A. 

David Goodwin, U.S. Department of Education, Chairperson 

Adriana de Kanter, U.S. Department of Education 

Brian Fu, U.S. Department of Education 

David J. Francis, University of Houston 

Gary Cziko, University of Illinois 

Yong Zhao, Michigan State University 

Patricia Green, RTI International 

ZhaoHong Han, Teachers College of Columbia University (attended Sunday only) 

Steven Stoynoff, Minnesota State University (attended Sunday only) 

China 

Wang Wenjing, Beijing Normal University 

Liu Dailin, China Central Radio and Television University (attended Saturday and Sunday) 

Zeng Dawei, National Center for Education Technology (attended Saturday only) 

Zhang Lianzhong, China National Institute of Educational Research (attended Sunday only) 

The Evaluation Planning Group (EPG) met as part of a larger meeting to discuss the 
development and evaluation of the Open Language Learning Initiative (OLLI), English language 
learning software that is being developed by the Hewlett Foundation and Coastline Community 
College. This project continues the efforts that were initiated as part of the CHENGO project, a 
joint partnership between the U.S. Department of Education and the Ministry of Education of the 
People’s Republic of China to create a highly innovative R&D product to test the effectiveness 
of using online technology to deliver foreign language instruction within a real-world 
environment, consistent with theory of language learning. In keeping with the agreements 
between the two nations, the U.S. Department of Education intends to evaluate the effectiveness 
of the software for Chinese middle school students. 

This document summarizes the key topics discussed by the EPG and presents their 
recommendations as well as concerns. 

1. Key evaluation question 

The EPG agreed that the evaluation should be designed to study the efficacy of OLLI, and that 
the key evaluation question is: 



7 Chengo is the name of the Chinese language learning program that was developed in China for U.S. 
middle school students. 
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Do students exposed to OLLI demonstrate increases in proficiency when compared to 
students without access to OLLI, given the same curriculum and under conditions that 
allow reasonable access to the software? 

The group recommended that proficiency gains be measured over the course of a year rather than 
a single semester. It was also recommended that a field trial of the evaluation be conducted in the 
fall of 2008, with implementation of the full-scale study in 2009. The group agreed that it was 
likely that there will be some technical issues and other problems when the program is first 
introduced in the fall of 2008 and mounting the full-scale evaluation at that time would be 
unwise. The group concurred that it would be best to conduct a field trial of the evaluation in the 
fall of 2008, to ensure that the data collection protocols and assessments are consistent with they 
way OLLI is implemented in China. 

2. Dimensions to be measured 

OLLI is currently intended to focus on the improvement of students’ speaking and listening 
skills; thus, the measurement of speaking and listening skills should be the primary focus of the 
evaluation. 

Reading and writing skills will also be assessed. These skills will be measured to determine 
whether (1) gains in speaking and listening enhance other language skills, or (2) gains in 
listening and speaking are made at the expense of gains in other areas. Reading and writing are 
emphasized in traditional classrooms, and skills in these areas are measured on local and national 
examinations. Thus, it is important to evaluate whether OLLI could have an unintentional 
detrimental impact by moving student effort away from these areas, or whether it might have a 
positive impact (or perhaps no impact) on reading and writing skills. 

At some point, the EPG will need to have more detailed specification from the developers about 
the subskills of each dimension so that tests can be identified or developed that parallel the 
language skills emphasized by the program. 

The EPG recognizes that OLLI may have an effect on student motivation to learn English, and it 
may also increase students’ awareness of U.S. culture. However, the EPG agreed that those 
constructs are secondary and will be included in study protocols as time and resources permit. 

3. Research design 

Given the evaluation question, a random assignment study is recommended. The EPG discussed 
assignment at the school level and at the classroom level. Because students are expected to 
sometimes work with OLLI outside of regular school hours, the possibility of contamination 
across classrooms appears to be high. Thus, randomization at the school level seems to be a more 
prudent option. One member of the evaluation group also suggested that it consider a smaller 
group of matched-pair schools, and work more aggressively in these schools to ensure that OLLI 
is fully implemented. 
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Textbooks and curriculum are probably major factors in students’ English proficiency. This 
suggests that schools be selected from only one or two districts, so that these factors can be 
controlled. 

The EPG will need the developers to specify minimum standards in a number of areas for 
schools to be included in the study. The standards should include the following: 

• How many hours per week do students need to have access to a computer to make 
meaningful progress using OLLI? 

• What are the minimum requirements for computers at participating schools? 

• Are there minimum requirements concerning teacher training in the use of OLLI? Do 
teachers need a certain level of experience with IT? 

4. Assessments 

The EPG discussed a number of existing assessments with members of the Language Learning 
and Pedagogy Working Group. Assessments that were discussed as possibilities included 
Cambridge University Young Learners English Test (Cambridge University), Maculaitis 
Assessment of Competencies II, Test of English Language Proficiency (Touchstone Applied 
Science Associates), and IDEA English Proficiency Test (IPT) 1 and 2 (Ballard-Tighe). Other 
assessments also mentioned as having unique features that might be useful included Phone Pass 
(Ordinate Technologies), the assessments developed for PANDA English (Beijing Normal 
University), Stanford English Language Proficiency Test (Harcourt Assessments), and the 
Woodcock Johnson Sentence Repetition subtest. (At a subsequent meeting, another test was 
discussed: the TOEIC Bridge). Problems were noted with each of the tests, including (1) some of 
the tests target TESOL standards rather than ACTFL standards, (2) some of the tests measure 
beginning English but content might be inappropriate because it is aimed at older or younger 
learners; and (3) some of the tests were developed for North American populations and may be 
culturally inappropriate for Chinese populations. 

The group also discussed using English exams developed for the Beijing municipality, or exams 
used in the districts where the evaluation will be implemented. These exams would provide good 
measures of reading and perhaps writing proficiency. They would need to be supplemented with 
assessments directed at listening and speaking. 

Developing specialized assessments for this project was also discussed. Some suggested that we 
could use the Cambridge Young Learners test as a model and adapt the subtests for the seventh- 
and eighth-grade students that will be targeted in the evaluation; the test is currently 
recommended for students from ages 7 through 12. Others mentioned developing sentence 
repetition tasks and perhaps a task asking students to narrate a series of pictures. The pros and 
cons of natural language samples were also discussed. 

The group concluded that a hybrid assessment would need to be used that incorporated 

• an existing assessment for use as an anchor and for providing context for the other 
measures; 
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• a speaking task that includes sentence repetition and picture narration as well as 
straight pronunciation; 

• a vocabulary assessment that uses the words that are common to both the national 
curriculum and the OLLI software (Both the comparison and treatment groups would 
have exposure to these words.); and 

• an existing assessment developed at the district level. 

The group also discussed the importance of the teacher’s level of fluency in English and 
potential ways it could be measured. One potential way would be to record a short interview with 
the teacher about participation in the evaluation study. 

5. Implementation of the evaluation 

The EPG recommended that the evaluation take advantage of the new equipment being given to 
schools through the Modern Distance Education Project. As described by Zheng Dawei, this 
project will equip 35,000 middle schools in the western region with new computers. These 
computers will be set up in a computer lab with Internet access. 

Twenty-five to 35 schools should be selected to be treatment schools, and the same number of 
schools should be selected to serve as the comparison group. 

Students in the first two years of middle school will participate in the evaluation. A decision to 
include primarily seventh-grade students or eighth-grade students will await more detailed 
information on the content of OLLI and selection of districts and an examination of the English 
curriculum in the district. 

Chinese scholars consistently expressed concern that teachers will be reluctant to devote 
instructional time to OLLI because of the importance of student performance on exams. They 
also were skeptical about the amount of time parents will allow students to use OLLI at home. 
One suggestion was that perhaps time in afternoon classes (devoted to IT or social practice) 
could be allocated to practice using OLLI. The EPG looks forward to learning more details about 
the teacher materials and courseware being developed for OLLI. 

The EPG suggested consideration of implementing the evaluation in boarding schools, where 
conditions could be more carefully controlled. In the western regions of China, many of the 
middle schools are boarding schools so that students do not have to travel great distances daily. 

If these schools have computer laboratories, it might be possible to arrange for students to have 
greater access to computers (at both comparison and treatment schools) than would be possible at 
day schools. 

The study will be designed to have a pretest and posttest. While the majority of the test will be 
administered to the whole class, individual assessments of student speaking will be necessary. 
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6. Roles and responsibilities 

It was understood that the Ministry of Education has volunteered to be responsible for 
identifying appropriate schools for the evaluation of the project. 

It will be necessary to identify the organization that will be responsible for providing the 
software to schools, along with orientation, training, and maintenance. The National Center for 
Education Technology may play this role. 

7. Next steps 

The United States will partner with an organization in China to conduct the evaluation. The 
Chinese partner will consult on the design of the assessment, review instruments to make sure 
that they are appropriate, and collect the data at the schools. 

It would be useful to obtain more information about the districts, schools, and students that might 
be selected to participate in the evaluation project. We would like data on numbers of schools, 
numbers of classes in schools, and number of students within classes. It would also be useful to 
know when students currently begin English language instruction in these districts. Also, if data 
are available, it would be valuable to obtain data from recent districtwide English examinations. 
These data would allow us to estimate the size of the typical gain in proficiency over the course 
of a year and would allow us to examine the variability within and between schools and classes. 
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GETTING STARTED 



PARTS OF A COMIC PANEL (SCREEN) 

This teacher aid is a reference for you, the teacher, so you can show your students the parts 
of The Forgotten World comic panels and their functions. This information can also be 
found in the Teacher's Guide. 



Log into The Forgotten World by typing 
in the URL 

http://21xp.com/enalish 

In the beginning, students will type 
in their Username and Password 
and then click the Sign In button. 




Students will see the Loader icon 
when the game is loading. Tell the 
students that they should wait until 
this icon goes away and the panel 
loads. 
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Below, is a typical comic panel (screen) in The Forgotten World. There are many parts to 
the panel that students should become familiar with so they can feel confident navigating 
the game. These parts are highlighted in red and numbered here, but each part separately 
will be described separately on the next three pages. 




This may be why he 
w is missing. ^ 



I didn't pack this, 
but it seems familiar 
v, somehow. ^ 



I / L-- Jonesy has 
pictures of :'r:S r:'s 

workshop. 
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Welcome, student! :+) 



1 . 



2 . 



3 . 



4 . 



5 . 



The Welcome Student text in the 
upper right-hand corner of the 
panel indicates that the student is 
logged into The Forgotten World. 



This may be why 
is missing 




/ he 

: I 



The Logout button in the upper 
left-hand corner of the panel is 
where students can click to log out 
of the game. 




I n the lower left-hand corner of the 
panel, students will see the 

episode/page indicator. This will 
tell them what episode and what 
page they are on. 




ep 2 - Comic Page 13 



Once every dialog bubble on the 
page plays, the students may move 
on to the next comic panel by 
clicking the Next button. Students 
may also go back to the previous 
comic panel by clicking the Back 
button. 






When the students roll over a 
dialogue bubble with their mouse, 
they may see certain words 

highlighted in green. These green 
words can be found in the 
Dictionary. 




I wish 
out 
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6 . 



7 . 



When students see that a part of the 
screen is outlined in blue, that 
means a game is present. Students 
should click on the glowing blue 
area to launch the game. 



Game is outlined 
in blue 




" Hungry F* 1x^-0 

lr> need oF supplies? 
k AC Hank's Superstore there's 
everything you could possibly 
need/ And For Coday only# iF you're 
Che Firsc caller Co answer Ch is 
riddle# you can win FREE sCuFF/ 
Just call 

555-0123 with your answers. 



I n the lower right-hand corner of the 
screen, students will see the PDA. 
Students may click on the PDA to 
open the PDA menu. 




Inside the PDA menu, students will see 
icons: 

■ Episodes 

■ Dictionary 

■ Game 

■ Help 

■ Credits 




Episodes Dictionary 




Games Help Credits 



If students click on the Episodes icon 
on the PDA menu, they will see a list of 
the episodes they have already 
entered. They will not be able to see 
the episodes they have not entered. 
Students can go back to the episodes 
they have entered by clicking on an 
episode icon in the list. 
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Students may click on the Dictionary 
icon on the PDA menu to open the 
dictionary and search for words. They 
can read the definition, listen to the 
word aloud, and listen to the word 
phonetically. 



Students can navigate to a particular 
game by clicking the Game icon in the 
PDA menu. 





Students can watch a short tutorial 
about how to navigate The Forgotten 
World by clicking on the Help icon in the 
PDA menu. 
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TECHNICAL FAQ 

FREQUENTLY ASKED QUESTIONS 

1. After pressing Next, I get a gray Zuka image and nothing happens. What do I do? 

This may be due to a problem connecting to the front-end, and/or back-end server. 

a) Note the current page number on the bottom left of the screen (Ep X, Comic Page X) 

b) Close the browser and open a new one 

c) Navigate back to the application 

d) Log in 

2. After playing a game, I get a gray Zuka image and nothing happens. What should I do? 

Please follow FAQ #1 

3. I want to play a game. How do I do that? 

a) Click on the PDA Icon at the lower right of the screen 

b) Click on the Games Icon 

c) Select the desired game 

4. How do I get The Forgotten World to be full-size on my screen? 

a) Press FI (function key) 

I want to play a specific game that I've already encountered. How do I do that? 

b) Click on the PDA Icon at the lower right of the screen 

c) Click on the Games Icon 

d) Select the desired game 
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THE FORGOTTEN WORLD STORY SUMMARIES 

The complete Teacher’s Guide, scripts for each episode and student worksheets are found 
on the enclosed CD-ROM. 

EPISODE 1: FLIGHT TO AMERICA 



Story Summary: We meet Uncle Jonesy, a scientist, as he is packing up his laboratory. He 
and his niece, Lia, are returning to the United States of America. Lia has made friends with 
one of your students, and she has invited him/her to accompany her to the United States. 
The student has arranged to meet Lia at the airport and board the plane to Los Angeles. 
While boarding the plane, the student is exposed to both formal and informal greetings. 
During the flight, Lia invites her friend to play video games. 

SHE • “ ^jnSjonesy^^SS^M^SjALilJT’ftkc. JonesyJ# 

iCo Jonesy 

SSWIMS, LiaBl miililUH. 



^jp#ih^wi'«]So sssmh], 

EPISODE 2 AND 3: LOS ANGELES 



Summary: Lia and her friend (your student) arrive at Los Angeles International 
Airport, but they cannot find Uncle Jonesy anywhere. They decide to go to his home, but 
Uncle Jonesy is not there. They call his assistants, Dave and Ray, to help them. While 
unpacking, Lia discovers a mysterious round ball-Zuka. When touched, it tries to 
communicate with them via symbols. Ultimately the players discover that Zuka wants 
them to go to San Francisco, but how will they get there? In Episode 3, Grace arrives and 
says she will drive everyone to San Francisco in her Recreational Vehicle (RV), but first 
they need to stop for supplies. 

SHE: 

LiajR H - IfiSinSJSHc^iiJ Jones yJ^J# 

° ftkit, fS Jonesy S' SiC o 

SS S % S S M ¥Da v e JR Ray ff BJj S If] . JT Jf ft $ fcf , M M ffi 

EfJlMliH — Zuka 

SWA MSjjSBS # SHi AXI o §Jo Sfn^fffiZuka^liSinS 

iH^Lh, lasings m M)im? 

SH3HS, Grace jTJlt T > ffoasstjft Pfl S (RV) , ^ X U 3 \ S HI 

. IB# 
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EPISODE 4: SAN FRANCISCO 



Summary: Lia, Ray, Dave, and Grace finally arrive in San Francisco. Zuka tells them to find 
"The Harmony Shop" and Dr. Hu. Dr. Hu gives them more history about Zuka, teaches them relaxation 
exercise, and then sends them on the task to collect potion bottles that will keep them 
safe from their enemies— the Creepers. 

Lia, 

Ray, DavefPGrace — f? IH # ill ° Zuka prijFfaTjy {til {H 7J ‘ AR iti # /A ” A 

EPISODE 5: DENVER, COLORADO 



Summary: The group is being led by Zuka across the Rocky Mountains to Denver, Colorado. 
They hope that Jonesy is there waiting for them. Once there, they go to NOAA, a national 
weather station, and then the Creepers arrive on the scene and the gang eludes them again. 
The gang searches for the key. 



ME: 

Lia— 'if ^ZukaifHJA AjAv'&Slil iAiOA# A A A{^TfJ 0 {til if ]# M Jones 

yAM^lAMA -7JM, 

#A;At!l7J7^Mo {MIX— 

7 #a^c 

EPISODE 6: ROSWELL, NEW MEXICO 



Summary: As the group heads for Roswell, New Mexico, Dave explains some of the rumors 
surrounding the city. After passing through security, Zuka leads everyone to a cave, where they 
find and release Uncle Jonesy. Before they escape, he explains about the earth’s “life force” 
energy and that the Creepers seek this energy. 

ME: A A AIA IrJ UtM® AAI A 1 A AASIAPt , Dave 
7 g o ZukaiEAiC A®J-AlIj 

M, {tii{nA3Pa7c7JA»it7Jonesy7(7(o A{M]MBt , {til M # A A it 
bj< 7j“ A A A ” 7J 'M , Ail^A^tizAATc&f^fuMo 
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EPISODE 7: DALLAS, TEXAS 



Story Summary: The group travels to Dallas and attends a Rodeo. The Rodeo is not quite 
right. The weather conditions continue to be out of the ordinary. Information is gathered 
that the Creepers are building a water capture machine. 

EPISODE 8: NEW ORLEANS, LOUISIANA 



Story Summary: The group goes to New Orleans. They arrive in the middle of a Mardi Gras 
celebration. Jonesy tells them about strange activities by the local river. The city is full of 
Creepers! 

Lia^tfH AAif^/K iMl AITXfciTlEHt Jonesy^ 

A A o AM ! 

EPISODE 9: DETROIT, MICHIGAN 



Story Summary: The group arrives in Detroit to investigate an abandoned car factory. As 
they heard in New Orleans, the city is teeming with Creepers. As they investigate, they find 
what the Creepers are doing there. 



mm : 

&t5jt#A2£ • ji&ras ’ a - 
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EPISODE 10: CHICAGO, ILLINOIS 



Story Summary: The group continues to chase the Creepers. Their next destination is at 
the Sears Tower in Chicago, Illinois ("The Windy City"). Zuka is trapped in the tower. They 
save Zuka, but he is very sick. They must harness the Earth’s life force to heal Zuka. This life 
force is at four power centers. 



SifisE: LiaiM)— A^o 

ir^vMc Zuka 

fiMn#gi)7zuka, ififtkiSf# 

ukai^io g H y h7fm4 ;, 7'o 

EPISODE 11: WASHINGTON D.C. 



Story Summary: They find out that Washington D.C. has important clues and race to get 
there before the Creepers do. Zuka feels better after touching water which becomes one of 
the four power centers. As the journey continues, Lia learns more information about her 
family history. 

lift: 

3\ m m o mm 

mmmtmmm, 55 ^ wit 51. 



EPISODE 12: NEW YORK, NEW YORK 



Story Summary: The group goes into New York City to find the other three power centers. 
Zuka transforms into a beautiful being with more power. They go to the Statue of Liberty to 
synthesize the powers and defeat the incoming Creepers, saving the world! 

tilt: 

ZukaffgJ Ilk ill 

xwtAi, #mtm 

W ! 
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THE FORGOTTEN WORLD GAME DESCRIPTONS 

Descriptions of the games that occur in the Forgotten World are listed below in 
alphabetical order, along with their occurrences within the episodes. These 
descriptions can also be found in Appendix A in the Teacher's Guide. 

Floating Phrases Game: The challenge for players is to construct valid sentences from 
a series of phrases that are floating across the screen. The players will need to understand 
the meaning of the words as well as the sequences in which the phases must be placed. 

Episodes: 1,5, 11 

Directions: Drag the phrases to the empty spaces at the bottom of the screen. Put three 
phrases in the correct order to make an English sentence. When all of the spaces are filled, 
click the Submit button to check your work. To win the game, make five different sentences 
before the time runs out. 



^ fe @ Sc ffl IE f# &} Jll Ff iffi tr W M M ^ , 

(Submit) IE^ o tu f: M aE fEI 2£ 
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Map Game: The player’s challenge is to guide other characters in the story through a map 
or maze to collect items and, ultimately, to reach a final target destination. The player does 
this by constructing a series of directions from a set of words and receiving written and 
audio feedback from the character who is receiving the directions. 

Episodes: 4 , 6 , 8 

Directions: Give Grace the directions she needs to find the bottles of potion. Click on one of 
the tabs at the top of the screen to select the type of sentence you want to use. Then, select 
words from the drop-down menus to complete the sentence. Click on the green button to 
hear your sentence and give the instruction to Grace. Avoid the creepers or you will lose 
the bottles you've collected. After you collect five bottles of potion, you can help Grace find 
the clue or you can collect more bottles to earn more points. Find the final clue to win the 
game. 

&G r a c e & tU fa $ 3 \ M ^ M W J b it o * * B * ± 7t S ^ ® U 5ft 

, ^^Grace^tM^o jfc 

M M # U toHsJlffiSjGrace^ $ 0 m M , 



San Francisco, California Map Game (Episode 4) 
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Roswell, New Mexico Map Game (Episode 6) 




New Orleans, LA Map Game (Episode 8) 




A Study of a Gaming Approach to English Language Learning for Middle School Students B-21 


















Appendix B - The Forgotten World Quick Reference Guide 



Phoneme Game: In the phoneme game, Zuka’s memory banks have become scrambled. As 
a result, Zuka has forgotten how to pronounce certain English words. The player must 
restore Zuka’s memory banks and its ability to communicate by unscrambling its memory 
and teaching it how to pronounce a series of words. 

Episodes: 9, 12 

Directions: Arrange the sounds correctly to teach Zuka how each word is pronounced. 
Click on the Audio button on the left to hear the sounds. Drag the sounds to place them in 
the correct order for each word. Complete the sound arrangement correctly for all five 
words to win the game. 

ft© s'JHSifff FpJtiStZuka® tiBJKlSIS'a M ifr & fill (Audio) 

IE 

amutiosmmtTi 
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Shopping Spree Game: The game is designed to test listening comprehension in a 
simulated, real-world environment. The player’s challenge is to locate a variety of items in 
a store by listening to audio clues, which are presented as snippets of conversations from 
different areas of the store. 



Episodes: 3 , 8 



Directions: Use the arrow keys on the keyboard to locate the items on your shopping list. 
Listen to the clues to help figure out where each item is located. When you find an item on 
your list, click on it and then check off the item on your list to add it to your shopping cart. 
To win the game, find all five items on your list before the time runs out. 



itl= if , A i±r m, 

tib f: HI 'If A hk Ml ip- iff , 



Shopping Spree (Episode 3) 
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Shopping Spree (Episode 8) 
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Story Recap Game: Designed to test reading and writing comprehension by asking players 
to briefly summarize key events they have encountered throughout the story. The player 
listens to the dialog for each event and then places the series of events in the sequence in 
which they occurred in the story. 

Episodes: 3, 6, 9, 12 

Directions: Click on the pictures to hear what each person said during the story. Then, put 
the words in the correct order to match what the person said and click the Done button to 
check your work. When all five people have the correct words, drag the pictures to place 
them in the order that these things happened in the story. 

M ^ ® n/f Wl ♦ ^ M A ^ XA o M m R S JH % ift M F? W 
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Tri Words Game is used as the pre-assessment. The game allows players to test their 
understanding of a wide variety of vocabulary by matching spoken audio to written words 
and phrases, as well as to symbols and images. 

Episodes: 1, 7, 11 

Directions for Pre-Assessment: Your goal is to clear the board of all triangles by matching 
sets of three or more triangles that are next to each other. To create a match, click on a 
triangle to see its Mandarin word. Then, find the word in English as it moves across the 
screen and drag it to the correct triangle to create a match. Match sets of three or more 
adjacent triangles to clear them from the board. Clear the entire board to win the game. 



M /A £ M ¥ ± 'M ^ in Jp- ih] ^ ^ U in in] in E 

f# ^ 1? It H c 

Directions: Your goal is to clear the board of all triangles by matching sets of three or more 
triangles that are next to each other. To create a match, click on a triangle to hear its word. 
Then, find a matching word or picture as it moves across the screen and drag it to the 
triangle. Match sets of three or more adjacent triangles to clear them from the board. Clear 
the entire board to win the game. 
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Zuka Symbol Puzzle Game: The player must communicate with Zuka using 
symbolic icons that represent words and concepts to construct a series of English 
phrases. It encourages comprehension of English by "constructing" communication G 

Episodes: 2 , 4 , 10 
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Directions: Click on the buttons at the left of the screen to hear each sentence. Then, 
drag the tiles from the top of the screen into the empty spaces to create each sentence in 
symbols and words. When all of the spaces are filled, click the Submit button to check 
your work. If any sentences are not correct, move those tiles to the correct positions 
and click Submit again. To win the game, use all of the tiles to complete all five 
sentences correctly. 

(Submit) ! 
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HOW TO ACCESS FILES ON THE TEACHER'S HUB 

tfwteachers.com 

Teachers will find the following information on the Teachers’ Hub: 

□ Class Progress Reports 

□ Individual Student Progress Reports 

□ The Teachers Guide 

□ Simplified Scripts of The Forgotten World Episodes 

□ Sound Clips from Episodes 

To go to The Forgotten World Teachers’ Hub: 

Step 1: Open a web browser and type in http: //www.tfwteachers.com 

Step 2: Log in with your user name and password. 




Step 3: Click on "My Class”, and view your students’ progress. 
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Mr. Lee's 7th Grade English Class 



Student Progress: 
Each box represents 
1 of 12 episodes. 



Spates* 

ACHitfK*T><4? 

BLewis189 

CPlX±ii»853 

SWngtn792 

CWalaceTM 



| Complete ^ In Progress 



j I -complete 
I I 



ftiSSs/l 

Date HI* »J* SSSBjS ffijRHH MSt iEM* 




Step 4: To see only one student's progress, click on that student name in your class list. 




Teacher22 

My class 
My account 
Log out 
Downloads 



Mr. Lee’s 7th Grade English Class Alex Chisholm 

« Go Back 



Alex Chisholm 



| Complete 



j In Progress □ Incomplete 



=c 



□c 



□c 



: Continue episodes and augment with worksheets 






Date 




HI* 




isas/s 


mmsBtifi] 




3E«« 


7/16 


AChisno!m442 


Episode 2 


Symbol Puzzle 


Home - Understands Words/Phrases (Prepositional) 


0:30 


14 


75% 


7/15 


AChishoim442 


Episode 2 


Comic 


Family/Friends/Home - Listening/Reading Words/Phrases 


1:24 


0 


n/a 


7/14 


AChishoim442 


Episode 1 


Floating Phrases 


Family/Friends / Reading Words (Past Tense) / Phrases 


2:40 


0 


75% 


7/13 


AChishoim442 


Episode 1 


Tn- Words 


Pretest Understands Words/Phrases 


0:15 


3 


30% 


7/12 


AChishoim442 


Episode 1 


Comic 


Greetings • Listening/Reading Words and Phrases 


1:13 


0 


n/a 
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Step 5: To download copies of the Teachers Guide, scripts, sound 
files or other teacher content, click on Downloads in the menu. 




Step 6: When finished, please click “Log OutG' 



Teacher22 



My class 
My account 
Log out 
Downloads 



Distributed by: 

Coastline Community College 

11460 Warner Avenue 

Fountain Valley, CA 92708 

Telephone: (714) 241-6109 

Fax: (714) 241-6286 

E-mail: CoastLearning@coastline.edu 

Copyright 2009 by the Coast Community College District 

Permission to use The Forgotten World Teachers Guide is granted under a Creative Commons 
Attribution Share Alike 3.0 United States License. 

If there are any questions regarding use and/or additional teacher resources, contact Coastline 
Community College (714) 241-6109. 

Funded in part by a grant from the William and Flora Hewlett Foundation. 

Coastline Community College District 

Trustees 

Mary L. Hornbuckle Walter G. Howald Jim Moreno 
Jerry Patterson 
Lorraine Prinsky 

Interim Chancellor Ding-Jo Currie, Ph.D. 
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Coastline Community College 

Evangeline N. Meneses, Ed.D., Acting President 

Dan C. Jones, Executive Dean, Center for Instructional Systems Development T ed Boehler, 
Ed.D., Dean, Learning Technology & Innovation 
Laurie R. Melby, Director of Video & Media Production 
Lynn M. Dahnke, Director, Marketing & Partnerships 

Robert D. Nash, Director, Instructional Design & Faculty Support Judy Garvey, Director, 
eMedia & Publishing 

Dave Thompson, Director, eLearning Research & Development 

OLLI Team 

Coastline Community College Chris Mefford 
Michelle Nieman 
Jingfang Satow 
Meg Yanalunas 

Learning Games Network Alex Chisholm 
Sherri Wright 
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Categories of 
Information 


ACTFL Oral Proficiency 
Interview 

OPI; OPIc; OPIc Jr. 


IPT 2004 

(IDEA Proficiency Test) 


The MAC II 

(Maculaitis Assessment of 
Competencies II Test of 
English Language 
Proficiency) 


Maker and Web 


ACTFL 


Ballard Tighe 


Questar Associates, Inc. 


site 


http://www.actfl.org/i4a/pages 


http://www.ballard- 


http://www.questarai.com/catal 




/index.cfm?pageid=3642 


ighe.com/products/la/ 


og_products/mac2_main.htm 


Level of English 


• OPI: 10 proficiency 


For Oral Tests: 


For testing: 


assessed 


ratings from Novice Low 


• Pre-IPT (ages 3, 4, and 5) 


• Red (grades K-1) 




to Superior 


• IPT 1 (grades K-6) 


• Blue (grades 2-3) 




• OPIc: 9 ratings from 
Novice Low to 


• IPT II (grades 7-12) 


• Orange (grades 4-5) 

• Ivory (grades 6-8) 




Advanced. 

• OPIc Jr.: 6 ratings from 


For Reading and Writing 
Tests: 


• Tan (grades 9-12) 




Novice Low to 


• Pre-IPT (ages 3, 4, and 5) 


For OLLI, Blue or Orange 




Intermediate High. 
(According to the ACTFL 


• IPT 1 (grades 2-3) 

• IPT 2 (grades 4-6) 


levels seem appropriate. 




Testing Office, OPIc Jr. 
is designed because of 


• IPT 3 (grades 7-12) 


English Competency Levels 
and standard score are 




the demand from the 


For OLLI, IPT 1 seems 


reported. E.g. For the speaking 




Asian market. It is 


appropriate for the oral test. 


test: 




currently used in Korea 


For the reading and writing 


• Basic beginner (score 




and the self-administered 


tests, IPT 1 or IPT 2 seem 


below 143 for orange; 




background survey that 
precedes the test is in 
Korean. A US version is 
still in development and 
its expected release is in 
November 2008) 


appropriate. 


below 153 for blue) 

• Beginner (1 43-1 62 for 
orange; 1 53-1 73 for blue) 

• Low intermediate (163- 

1 85 for orange; 1 74-1 94 
for blue) 

• High intermediate (186- 
21 5 for orange; 1 95-220 
for red) 

• Advanced (above 21 5 for 
orange; above 220 for 
blue) 


Skill areas 


Speaking and oral proficiency 


1 . Oral: both speaking and 


1. Speaking (individual); 


assessed 1 




Listening (individual); pre- 
recorded examiner is not 
recommended because 
the test requires 
interactions. 2 

2. Reading (group) 

3. Writing (group) 


examiner to rate 
responses immediately 

2. Listening (group); 
audiotapes can be 
purchased for the Blue, 
Orange, Ivory, and Tan 
levels 

3. Reading (group) 

4. Writing (group) 



Continued on next page. . . 
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Categories of 
Information 


ACTFL Oral Proficiency 
Interview 

OPI; OPIc; OPIc Jr. 


IPT 2004 

(IDEA Proficiency Test) 


The MAC II 

(Maculaitis Assessment of 
Competencies II Test of 
English Language 
Proficiency) 


Potential to 


• Only proficiency ratings 


• Alternate test forms to 


• Alternate test forms are 


measure growth 


are offered but the 


ensure reliability of scores 


available at the testing 




ACTFL Testing Office 


between testing (Forms E 


level for pre- and post- 




claims that OPIc is used 


and F for oral tests and 


testing 




regularly to measure pre 


Forms C and D for 


• Raw score and converted 




and post semester 


reading/writing tests) 


standard score can be 




testings 


• The IPT oral tests 
provides score levels (raw 
scores can be derived) 

• The IPT reading tests 
generate percentiles, 
Normal Curve Equivalent 
scores, and Standard 
scores (raw scores can be 
calculated) 

• The IPT writing tests use 
a four-point rubric for 
scoring 


tallied for each student 
and for each skill area 


Time required 


30-40 minutes (20-30 


• Oral (individual): untimed, 


• Speaking (individual): 6 


for 


minutes for the test and 1 0 


but 15-20 minutes, scored 


min 


administration 


minutes for background 


as students are being 


• Listening (group): 20-30 




survey and self-assessment) 


tested. 

• Reading (group): untimed, 
but usually 60 minutes 

• Writing (group): untimed, 
but usually 35 minutes 


min 

• Reading (group): 35-50 
min 

• Writing (group): 15 min for 
blue; 40-45 min for higher 
levels 


Special 


OPIc and OPIc Jr.: Web 


The oral test requires a live 


Speaking test requires a live 


requirements for 
administration 


accessibility and high speed 
connection, headsets with 
microphones 


examiner. 


examiner. 


Appropriateness 
for age group 


OPi and OPIc: General 
audience 

OPIc Jr.: K-8 students 


Pre-Kto 12 


Pre-Kto 12 



Continued on next page. 
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Categories of 
Information 


ACTFL Oral Proficiency 
Interview 

OPI; OPIc; OPIc Jr. 


IPT 2004 

(IDEA Proficiency Test) 


The MAC II 

(Maculaitis Assessment of 
Competencies II Test of 
English Language 
Proficiency) 


Cultural 


• The Asian version of the 


Based on the sample tests, 


Based on the sample test, 


appropriateness 


OPIc Jr. lets the testers 


two issues were identified: 


three issues were identified: 




identify themselves in 


• Store varieties: Pet store 


• Food items: bread and 




four school levels: 


• Children’s activities: boy 


milk 




Preschool/Kindergarten, 


goes to the post office, 


• Pictures: few Asians and 




Grades 1-3, Grades 4-6, 


eating popcorn and lying 


some everyday objects 




and Middle School. The 
U.S. version of the OPIc 
Jr. is being released at 
the end of this year, but 
the school levels are 


on the floor in front of TV 


may look different in China 
(e.g. a school desk) 

• Person names: some 
Spanish names 



Elementary, Middle, and 
High schools. 

• Based on the (demo) 
Asian version of the 
OPIc Jr., the background 
survey is geared toward 
the Korean audience 
(e.g. mention of after- 
school academy; 
activities), but the ACTFL 
Testing Office claims that 
they can make 
adjustments and also 
make the Chinese 
instructions available. 

• Some test questions are 

asked in a row. The 
ACTFL Testing Office 
explains that the test 
does not intend for the 
test-taker to be able to 
answer all the questions. 
Their proficiency is 
measured by the details 
that they can provide at 
any of the questions 
read, even when they 
say, “I can’t answer the 
rest of the questions 
because I did not 
remember what they 
were.” Such a statement 
may demonstrate their 
oral proficiency. 



Continued on next page. . . 
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Categories of 
Information 


ACTFL Oral Proficiency 
Interview 

OPI; OPIc; OPIc Jr. 


IPT 2004 

(IDEA Proficiency Test) 


The MAC II 

(Maculaitis Assessment of 
Competencies II Test of 
English Language 
Proficiency) 


Cost 


Testing, scoring, and 


Testing: 


Testing: 




customer support: 


• 


In-service training kit for 


• Classroom set for a class 




• $25 per test for OPIc Jr.; 




IPT: $135 


of 25: $1 80 for red/blue; 




volume discount is 


• 


IPT 1 oral test set (50 


$239 for higher levels 




available 




booklets): $185 


• Administration manual: 




• $50 per test for OPI 


• 


IPT 1 oral test alternate 


$45 








forms (50): $51 


• Test booklets: $105 per 25 






• 


IPT 1 oral test diagnostic 


for red/blue; $138 per 25 








answer sheets (50): $51 


for higher levels 






• 


IPT 1 oral test scannable 


• Pictures booklet (red/blue 








answer sheets (50): $51 


only): $18 






• 


IPT 1 reading/writing test 


• Answer sheets (orange, 








set (50): $228 


ivory, tan): $21 per 25 






• 


IPT 1 reading/writing 


• Writing forms (orange, 








alternate test booklets 


ivory, tan): $29 per 25 








(50): $81 for reading and 


• Listening test audiotapes: 








$60 for writing 


$27 










• Student record forms: $32 



Scoring: 

• Hand score 

• Use own scanning 
equipment 

• Contract with a scanning 
service 



per 25 

Scoring: 

• Scoring services and 
rating of open-ended 
writing items 

o Minimum charge: 

$250 

o Per student: $3.62 for 
red/blue; $2.57 for 
orange, ivory, or tan 

• Hand scoring 

o EZ Converter: $65 per 
form 

o Scoring keys: $19 
o Handbook with norms 
tables: $49 



Option to adapt 
instrument 



Vocabulary 
overlap with 
OLLI 



OPIC Jr. 

• The background survey 
determines the test 
questions. For the OLLI 
audience, a uniform 
background survey 
needs to be used. 

• A button is available for 
the test takers to repeat 
the question once, but 
they cannot go back to a 
previous question. 

OLLI is developed based on 
ACTFEL Standards and 
ELLS E-Learning Standard 
(to raise speech acquisition 
from Novice Low and Novice 
High) 



According to psychometrician TBD 

George Seretis of Ballard 

Tighe (Maker of the IPT), 

minor adaptation is possible as 

long as the test item remains 

in context. However, if the 

change is likely to change the 

item, it is better to simply skip 

it and give the student the 

credit. 



To be assessed 



To be assessed 



Continued on next page. . . 
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Continued from previous page. 



Categories of 
Information 


ACTFL Oral Proficiency 
Interview 

OPI; OPIc; OPIc Jr. 


IPT 2004 

(IDEA Proficiency Test) 


The MAC II 

(Maculaitis Assessment of 
Competencies II Test of 
English Language 
Proficiency) 


Vocabulary 


? Need curriculum materials 


To be assessed with 


To be assessed with 


overlap with 
curriculum 




curriculum materials 


curriculum materials 


Estimated effect 
size over one 
year 

1 AA-rr, 7T— 


Information unavailable 


According to psychometrician 
George Seretis, they did a 
study five years ago but no 
information is available on 
standard deviation. For the 
oral test, the IPT 1 (level 
recommended for OLLI based 
on preliminary assessment) 
has one score level gain 
(about 14 points in raw score). 
For the reading and writing 
tests, there is a mean of 1 1 - 
point gain and three-point gain, 
respectively. 


TBD 



1 ACTFL does offer a test to assess writing proficiency, but it is intended for the general audience. 

2 It’s possible to derive speaking and listening scores separately, but the norms have not been updated 
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I. Background and Goals for the Pilot 

In the fall of 2008, the Chinese Ministry of Education informed the U.S. that schools in Gansu 
Province would be available to participate in the evaluation of The Forgotten World (TFW), a 
computer-based game designed to help middle-school students learn English. A brief trip was 
made to Gansu Province in December, 2008, to visit schools and assess whether these schools 
would provide a good base for the evaluation. Among our questions were the following: 

• Are schools and teachers willing to implement the program as a supplement to regular 
instruction, providing access to TFW in the computer lab at least one hour per week? 

• Do the students in seventh or eighth grade exhibit the expected level of English that is 
targeted by TFW? 

• Do the schools have the requisite computing configurations to implement the program? 

We found that principals and teachers were eager to implement the program, and teachers in the 
two pilot schools that were visited promised that students would be given access to TFW. They 
also asked for a teacher’s guide and some ideas about how they might use TFW in class. 

Students in all grades (7 - 9) were eager to participate in the pilot sessions and, based on the 
preliminary tests and examination to test booklets, had sufficient background in English to 
benefit from TFW. Unfortunately, the computer labs did not have the suggested hardware and 
software. Staff from the Provincial Center for Educational Technology (PCET) were able to 
develop a “work-around” for our visit that supported most program features; however, upgraded 
equipment will be needed for schools to fully use the program. Needed upgrades to the 
configurations will be made for the pilot test (two schools) but are uncertain for the full-scale (10 
schools). 

We are proceeding with the pilot test and planning for the full-scale evaluation. The purpose of 
this document is to outline our goals for the pilot test. The aim of the pilot is to ensure that all 
instruments and procedures are tested in advance of the full-scale evaluation. We have five 
primary goals for the pilot: 

1. Confirm that all instruments are culturally appropriate for Chinese middle-school 
students and the proposed test battery can measure change over an academic year. 

2. Develop and collect data to measure program implementation. 

3. Assess whether teacher training is sufficient. 

4. Test all data collection protocols and procedures. 

5. Gather feedback from teachers and students concerning their experiences using The 
Forgotten World. 

We have developed specific evaluation questions to be answered under each of these goals. 
Exhibit D-l presents the evaluation questions, the method of data collection, the organization or 
person responsible, the timing of the data collection (April or June), and the organization that 
will enter the data and conduct the analysis. This document includes a section discussing each 
goal and related evaluation questions and plans. 
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Exhibit D-1. Evaluation Questions for Gansu Pilot 



Evaluation Questions 


Method/Source 


Responsible 

Person 


April Data 
Collection 


June Data 
Collection 


Analysis 


Gather information on assessments for design of full-scale evaluation 


Is proposed test battery 
able to measure change 
over one year? Proposed 
battery to include IPT 
reading, writing; Mac II 
listening (adapted); use 
pre- and post-test 
versions 


Assessments 
administered at two 
schools, all 7 th - and 8 th - 
grade students 


RTI 


Zhu 


Additional 
testing if 
needed 


RTI 


Pilot speaking test using 
Skype 


Sample of students 


RTI 


Green 


Green 


RTI 


What are students 
beginning and end-of- 
year scores on provincial 
tests? 


School or province 


ECNU 




Zhu 


ECNU 


Questionnaires 


Student and teacher 
questionnaires to gather 
information on prior 
experience with English, 
current exposure and 
demographic information 


RTI/ECNU 


Zhu 


Zhu 


ECNU 




Measure implementation of program 


Can we track student 
use? 


System data 


LGN 


LGN 




RTI 


How do teachers use 
TFW? 


Teacher logs 


RTI 


Green 


Green 


RTI 


Pace of learning - how 
much is covered? 


Teacher logs and system 
data 


RTI and LGN 


Green 


Green 


RTI 


Do teachers use Teacher 
Reports? 


System data and post- 
pilot quex 


RTI and LGN 




Green 


RTI 


What type of technical 
support was needed? 


Structured interviews 
with PCET and school 
tech personnel 


RTI/ECNU 




Zhu 


ECNU 


What problems did the 
school face in 
implementing TFW? What 
were the rewards? 


Structured interviews 
with principals 


RTI/ECNU 




Zhu 


ECNU 


Confirm hardware and 
software configurations 
used at schools 


Checklist completed by 
school tech personnel 


RTI/LGN 


Zhu 




ECNU 


Evaluation Questions 


Method/Source 


Responsible 

Person 


April Data 
Collection 


June Data 
Collection 


Analysis 


Evaluate teacher training 


Materials sufficient for 
startup? 


End-of-training 

questionnaire 


RTI 


Green 




RTI 


How much teacher 
training is needed? 


End-of-training 

questionnaire 


RTI 


Green 




RTI 


Feedback after pilot 
complete — what training 
would have helped? 


Post pilot questionnaires 
and focus groups 


RTI 




Sha 


RTI 



(continued) 
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Exhibit D-1. Evaluation Questions for Gansu Pilot (continued) 



Evaluation Questions 


Method/Source 


Responsible 

Person 


April Data 
Collection 


June Data 
Collection 


Analysis 


Test procedures for data collection 


How are test 
administrators identified? 
Can teachers be used? 


Protocols to be 
developed by ECNU with 
RTI approval 


ECNU 


Zhu 




RTI 


What is the best method 
for training test 
administrators? How long 
is needed? 


Protocols to be 
developed by ECNU with 
RTI approval 


ECNU 


Zhu 




RTI 


Do test administrators 
follow protocols? 


Observations of test 
sessions 


RTI/ECNU 


Zhu 




ECNU 


What is the best way to 
handle informed consent? 


Protocols to be 
developed by ECNU with 
RTI approval 


RTI/ECNU 


Zhu 


Zhu 


RTI 




Gather feedback on using The Forgotten World 


How do students use the 
Forgotten World? What 
do they like and dislike? 
How does it help them 
learn English? 


Questionnaires and cog 
interviews 


RTI 




Sha 


RTI 


What do teachers like and 
dislike about The 
Forgotten World ? 


Post-pilot questionnaires 
and cog interviews 


RTI 




Sha 


RTI 



1. Pilot test assessment instruments 

Pretest results 

In December, we conducted a small pretest of two instruments that were identified as promising, 

o 

the Mac II (grades 2-3) and the IPT Level 2. The Mac II was administered to 20 seventh-grade 
students. We administered the IPT to 20 eighth-grade students. A summary of results are 
presented in exhibit D-2. 



Exhibit D-2. Test scores for IPT and MAC from Gansu December Pretest 





IPT 


MAC 




Total items 


Range 


Median 


Total items 


Range 


Median 


Reading 


53 


30-45 


38.5 


35 


4 - 18 


11 


Writing 


6' 


2-6 


5 


35 


4-30 


12.5 


Listening 








24 


4 - 16 


9 


Speaking 















1 . Only one part of IPT Writing test was administered because of time constraints. 

The MAC appears to capture the range of English competency for seventh-grade students and the 
IPT captures the range of competency for the eighth-grade students tested at the two schools. The 



8 Other tests were reviewed but removed from consideration. The ACTFL proficiency tests and the TOEIC 
are geared for adult populations; the TOEIC Bridge was age-inappropriate; and the Cambridge Young Learners 
Exam does not provide scale scores (only proficiency levels). The distributor of the Stanford English Language 
Proficiency test was unwilling to work with the project 
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only subtest that appears to have a ceiling or floor effect is the IPT writing test; however, we 
only administered one subtest. Since we did not administer the most difficult section, we do not 
believe that this would be an issue in the full administration. 

We also matched the vocabulary used in TFW, the English textbooks used in the schools, and 
assessment items. For the assessments, we identified items that three or fewer students answered 
correctly. Only one item in the IPT fell into this category, and the key word in the item was 
“pain,” a word that is not covered until the ninth-grade textbook. The MAC contained 1 1 items 
that fell into this category: Seven of those key words could not be found in the textbooks, one 
was in the seventh-grade text, two in the eighth-grade text, and one in the ninth-grade text. The 
MAC would need substantial modification to be used in the evaluation. 

We also compared the vocabulary in the Chinese texts for seventh and eighth grades with TFW 
vocabulary to try to determine which grade would be better “match” for TFW. The TFW was 
designed for students with some prior exposure to English and assumes that students have a 500- 
word vocabulary. We matched the list of 500 words to TFW and present the results in 
Exhibit D-3. 



Exhibit D-3. Prerequisite vocabulary words presented in textbooks by grade level 



Grade Level 


Count 


7th grade 


269 


8th grade 


71 


9th grade 


23 



Of the 500 words, 363 could be found in the middle school textbooks, with over half (269) 
appearing in the seventh-grade text book. Thus, it seems unwise to assume that seventh-grade 
students will have the working vocabulary expected by the game designers. 

Exhibit D-4 presents information on matches between the TFW vocabulary and the middle 
school textbooks. In addition to the 500 prerequisite words, TFW presents over 1,000 new words 
across the 12 episodes; about half of the words (508) are also found in the Chinese middle school 
textbooks, and most of these (267) are found in the seventh-grade text. 

The seventh- and eighth-grade textbooks contain about 492 words each or 985 words total. This 
means that about 40 percent of the vocabulary ((267+145)/1000) normally taught in these two 
grades will be reinforced by TFW. 
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Exhibit D-4. Matched Vocabulary Count by TFW Episode and Textbook Grade 
Levels 



TFW Episode 


7th 


8th 


9th 


Total 


1 


65 


25 


11 


101 


2 


28 


5 


10 


43 


3 


9 


15 


18 


42 


4 


20 


30 


14 


64 


5 


11 


3 


2 


16 


6 


19 


14 


10 


43 


7 


16 


13 


5 


34 


8 


18 


14 


13 


45 


9 


31 


6 


6 


43 


10 


10 


4 


0 


14 


11 


27 


8 


1 


36 


12 


13 


8 


6 


27 


Total 


267 


145 


96 


508 



Suggestions for spring pilot 

Based on our experience in Gansu with the pretest, we suggest the following: 

1. We will ask eighth-grade students to participate in the evaluation. This group will have 
much of the prerequisite vocabulary and will have learned about one-quarter (267) of 
TFW new words in seventh grade. There will still be about three-quarters of the TFW 
vocabulary that they will have to learn through the episodes and games. Given that 
vocabulary is antecedent to more complex language competencies (sentence structure, 
grammar, etc.) we think that eighth-grade students are in the best position to benefit from 
TFW. 

2. For reading, we will use the IPT. In the pretest the reading test seemed well-matched to 
student abilities — there were no students who were at the top or bottom of the test. This 
test also requires very little modification, if any. We will need to examine the alternate 
forms of this test (we need one form for the pre-test and one for the post-test) to make 
sure that there are no culturally inappropriate items. 

3. For writing, we will use the IPT, including a second writing subtest that requires students 
to write a story. We will not use the sub test that measures “writing conventions” such as 
punctuation, since TFW does not aim to improve student competency in this area. The 
range of total possible scores on the writing subtest will be 0 to 9 points, although we 
could consider expanding the grading rubric. 

4. For listening, we propose to revise the MAC listening test. For the MAC test, a tape is 
supplied and a native U.S. English-speaking person provides instructions and reads all the 
questions to students. In administering the pretest, we found the instructions to be 
somewhat cumbersome and the layout of the answer sheets was confusing to students. 

We will simplify the directions and the answer forms. Two forms will be needed; one for 
administration at the beginning of the year and one for administration at the end of the 
year. 
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5. RTI will explore devising a new test, assessing reading and listening skills based on items 
that appear in TFW that do not appear in the school textbooks. This test will be 
administered at the end of the school year. The structure of the listening test and items 
will mirror the MAC listening test and will include: eight items that ask students to select 
a picture that matches the word, eight items that are based on two short stories that are 
read to students; and eight items that require students to select an appropriate response to 
a conversational question. An audiotape will be professionally recorded that will be used 
in all classrooms. There will be 26 reading items based on TFW. The total test will have 
50 items. 

6. The reading, writing, and listening tests should provide us with a scale of at least 86 
points (53 reading, 10 writing, and 24 listening), although we may want to consider 
weighting the subtests. 

7. For the pretest, we propose administering a speaking test to a subsample of students — IF 
the voice recognition applications can be made to work on the school machines. The 
speaking test will be patterned on the MAC and will have three parts: students view 
pictures and say the word in English; students are asked personal or conversational 
questions about themselves; and students view a picture and are asked to tell a story 
about the picture. The sessions will be tape-recorded. We had hoped to administer these 
tests via computer but do not believe that the Internet connections will be sufficient. 

8. We will ask to administer the assessments to ALL seventh and eighth grade students in 
the two schools participating in the pilot. This will provide us with about 1,000 students 
per grade or about 500 per test form. It will also allow us to estimate the growth in 
language during an academic year by comparing end-of-year seventh-grade students 
(equivalent to beginning eighth-grade students) with end-of year eighth-grade students. 
Note that this will only be an approximation since we won’t have beginning and end-of 
year scores for the same students. (We were unable to launch the pilot in the fall because 
of delays in software development). 

9. The proposed test battery will require approximately two hours of testing time. Reading 
typically requires 60 minutes; writing 30 minutes and listening 30 minutes. 

10. The IPT reading/writing test costs approximately $4 per booklet ($8,000 for the pilot). 

11. Schools will be asked to provide beginning and end-of-year test scores on the district 
assessments for eighth-grade students participating in the pilot. Officials in Gansu agreed 
to this in December. We will need to collect identifying information from students (name 
and school, perhaps birth date or school ID) to make the match. 

2. Measure implementation 

One of the goals of the pilot is to make sure that conditions are such that schools and teachers 
can implement the program. Thus we want to confirm that TLW is fully functional, the 
equipment is able to support student use, any technical support that is required is given to 
schools, and teachers refer to TFW in the classroom and give students the opportunity to use the 
program. At the December meeting in Gansu, the principals stated that they would make sure 
that students have access to the computer lab for 40 minutes at least once a week, and the 
principal of school committed to a higher level, promising to incorporate TFW in English classes 
once each week. We will want to confirm these agreements and monitor the implementation. We 
also want to learn how quickly or slowly students will move through TFW so that we can time 
the full-scale data collection to closely follow the completion of TFW. 
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Exhibit D-l provides information on the questions we want to answer in the pilot. We will need 
to collect data from several sources: 

• We have asked Learning Games Network (LGN) to collect information on students’ use 
of TFW (total time and highest-level episode), teacher use of TFW (total time or number 
of logins and a count of times teacher reports are accessed), and “down periods” when 
the system is not be used at all by the school (to uncover system issues or other events 
preventing the use of the program). PCET will be asked to contact schools that have 
down periods of more than one week to see if technical assistance is needed. 

• Teachers will be asked to keep brief logs of their use of TFW, both in class and when 
students are given access to the lab. We will develop a Chinese-language template for 
the log. 

• Structured interviews will be conducted at the end of the pilot with PCET and school 
tech personnel to document the types of system support that were needed. 

• Structured interviews will be conducted with principals at the end of the pilot to discuss 
any issues in implementation and also the perceived benefits of using TFW. 

• A checklist documenting the elements of the computing infrastructure will be collected 
from each school. 

3. Evaluate teacher training 

In the pilot, we want to find out from teachers if the training they received was sufficient for 
their subsequent use of TFW, and to gather their suggestions for ways it might be improved. We 
will do this at two points in time. An initial questionnaire will be developed to be administered at 
the end of teacher training to assess whether teachers feel they have the knowledge they need to 
begin using TFW. At the end of the pilot, teachers will be asked to complete another 
questionnaire assessing, in retrospect, if the training was adequate. These data will be supplied to 
Coastline to help in creating the final teaching materials for the full-scale evaluation. 

4. Field test data collection procedures 

Professor Zhu, our data collection partner in China at East China Normal University (ECNU) has 
arranged to bring graduate students from Shanghai to administer the tests and will also hire 
additional administrators in Gansu province (probably from Northwest Normal University). We 
will arrange a training session with Professor Zhu on the administration of the tests and will help 
to prepare training manuals. RTI will observe the test administrator training and some test 
administrations. RTI will also prepare a short information form that test administrators will be 
asked to complete that describes salient features of the test session as well as any issues that 
arose during the test session. 

5. Gather feedback on experiences using The Forgotten World 

At the completion of the pilot in June, we will collect some information concerning students’ 
experience using TFW on the student questionnaire. We will also collect information from 
teachers on a teacher questionnaire, as well as convene a focus group of teachers to discuss the 
experience. 
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We may wish to conduct some number of cognitive interviews with students as they use the 
software. This needs to be discussed with LGN and Coastline, to determine whether this 
information will be useful to future development. 

6. Pilot questionnaires 

Teacher questionnaires 

Teachers will be asked to complete a training evaluation, a teaching log, and a questionnaire, all 
of which will be prepared in Chinese. The questionnaire to be in administered in June will 
capture information on: 

• The quality of training 

• Experiences using TFW 

• Experience teaching English 

• Education and credentials for teaching English 

• Informal exposure to English, including travel 

• Likelihood that they will continue using TFW 

• Limited demographic items 



Student questionnaires 

Questionnaires will be administered to students in April and June. The questionnaires will be 
designed to capture: 

• Experience using TFW 

• Years of formal English instruction 

• Informal exposure to English (TV, videos, friends or relatives, etc) 

• Importance of learning English 

• Expectations for future 

• Limited demographic items 

RTI will develop a list of items and ECNU will develop the Chinese-language questionnaires. 
The questionnaires will be reviewed by a native Chinese-speaking survey methodologist at RTI. 

II. Plan for Full-Scale Evaluation 

There are three major questions to be answered by the full-scale evaluation: 

1. Do students exposed to TFW demonstrate improved English-language skills in reading, 
writing, and listening, compared to students who have not been exposed to TFW? 

Primary measure: Score on English-language tests at beginning and end of year that are 
developed by the province. 

Secondary measure: IPT reading and writing tests; MAC listening tests administered to treatment 
and comparison schools at beginning and end of the year. 
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2. Do students exposed to OLLI master the vocabulary and language presented by the 
program that is not covered by classroom texts? 

Primary measure: Reading and listening test developed for end-of-year based on words unique to 
TFW that are not included in eighth-grade textbook. This test would need to be developed by 
RTI and administered to treatment schools only. 

3. What implementation factors make a difference in the degree to which students benefit 
from TFW? 

Primary measure: Primary outcome measure will be TFW -based test; may also used provincial 
tests or IPT or MAC II. Implementation factors to be measured include: time spent on TFW, 
teacher’s use of TFW in the classroom, level of English at introduction of TFW, etc. 

RTI is currently planning the full-scale evaluation. On approval from ED, we will ask PCET to 
identify 20 volunteer schools. These schools must be willing to participate in the evaluation, use 
the Go For It! English textbook series, be located in reasonably accessible area, and have the 
requisite hardware. We will then randomly assign 10 as comparison schools and 10 as treatment 
schools. We will request that we randomly select four classrooms of eighth-grade English, taught 
by two different teachers. We were told that there are approximately 60 students per class, so this 
will provide over 200 students per school, or 2,000+ participants in the treatment and comparison 
groups. 

With 20 schools, assuming an intraclass correlation coefficient of .22, we will be able to detect 
an effect size of approximately .65. 9 This means that the treatment group will need to have a 
mean score that is two-thirds of a standard deviation higher than the comparison group to 
conclude that TFW is effective. Since we have no previous data on the effectiveness of TFW, it 
is difficult to know whether this assumption is warranted — it is an extremely high bar. 

Twenty teachers will need to be trained in the use of TFW. If teachers are asked to administer the 
tests, we will need to provide training on test administration to 40 teachers. At this point, we do 
not envision providing an alternative intervention to comparison schools. 

We may want to consider testing additional eighth-grade students who are not exposed to TFW 
at the treatment schools. If there is no contamination, these students would provide strong 
statistical controls. If there is contamination, it might provide documentation for the “viral” 
effect of gaming applications and provide us with data that we can use to examine dose-related 
response. 

Additional power calculations will be prepared once we have the scale for the assessments. 
However, final power calculations cannot be estimated until we have sample assessment data. If 
it is clear that we cannot meet the power requirements, we may wish to reconsider the first 
question posed above. We may also want to increase the prominence of the qualitative data 



9 This was estimated earlier assuming 70 students per school. We expect that increasing the sample size in 
each school will allow us to detect a slightly lower effect size. 
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(teacher focus groups and student cognitive interviews) in the final design for the full-scale 
evaluation. 

III. Summary and next steps 

Once this plan is approved, a subcontract will be issued to ECNU to conduct the tasks assigned 
in Exhibit D-l. We will also ask outside consultants to review the plan, and proposed 
instruments, and provide feedback by March 22. Instruments will be submitted to ED for 
approval on March 30. 
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Appendix E: 

Evaluation Plan Update, July 2009 (Revised) 
1. Purpose of the evaluation 



The purpose of this project is to evaluate the implementation and efficacy of English language 
learning software developed especially for middle school students in western China. Following a 
trip to China by the U.S. president in 2001, the U.S. and China signed a bilateral memorandum 
of understanding for a joint research and development project, known as the U.S. -China 
E-Language Project. The E-Language software — named The Forgotten World (TFW) — has been 
developed by Coastline Community College and Learning Games Network, with funding from 
the Hewlett Foundation. 

The focus of the evaluation is to compare the proficiency of the students exposed to the TFW 
program with the proficiency of students who do not have access to TFW. We will also identify 
implementation factors that facilitate or impede program effectiveness. A secondary purpose will 
be to obtain user reaction to instructional features of the program. The major research questions 
to be addressed are as follows: 

• Do students exposed to TFW English demonstrate increases in proficiency when 
compared to students without access to TFW English, given the same curriculum and 
under conditions that allow reasonable access to the software? Proficiency in two main 
areas and two cross-cutting areas will be measured: 

o Listening skills 
o Reading skills 

• Does TFW increase student motivation to learn English? 

• Does the program work for students at all levels of beginning English? 

• What implementation factors make a difference in promoting student proficiency? 

An experimental study, conducted by RTI in partnership with East China Normal University 
(ECNU), is being conducted to answer this question. Although the circumstances do not allow 
for a fully powered hypothesis-testing design, the current design will allow us to examine the 
impact of TFW on a large group of students and will allow us to generate hypotheses about 
whether the program is effective. 

The design will compare eighth-grade students in five comparison and five treatment schools in 
Gansu province, China. The selected schools are large middle schools with approximately 2,800 
eighth-grade students enrolled in the treatment schools and another 2,800 enrolled in the 
comparison schools. While we initially requested 20 schools (10 treatment and 10 comparison), 
the provincial authorities determined that there were not enough schools that met the 
technological and participation requirements for the project. A decision was made that we would 
use the 10 schools that met the requirements and divide them into two groups, a treatment group 
that would implement TFW and a comparison group. 
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Students in the second year of middle school (eighth-grade students) will participate in the 
evaluation. Proficiency gains will be measured over the course of a year (two semesters), and the 
evaluation will be conducted in the 2009-2010 academic year. A schedule is provided in the 
attachment. 

2. Instrumentation (Task 3) 

Assessment instruments and questionnaires were tested in two pilot schools in the province in the 
spring of 2009. The IPT Reading and IPT Writing assessments were used, along with the MAC 
II Listening assessment. The IPT Reading assessment was used without modification; the other 
two assessments were adapted based on preliminary work in China. For the writing test, some of 
the picture prompts were substituted using pictures from other IPT tests that were more culturally 
appropriate for Chinese middle school students. The listening test directions were modified so 
that the directions were delivered to students in Chinese, and the answer form was modified to 
make it more understandable to students. All of the listening instructions were professionally 
recorded at RTI to ensure comparability across sites. 

The purpose of the pilot was to make sure that the tests were able to measure growth in English 
proficiency among this population and that the modifications made to the tests did not alter the 
equivalency of the pre- and post-test forms. The tests were administered at two pilot schools to 
seventh- and eighth-grade students. Since we did not have a full year for the pilot, end-of-year 
seventh-grade students represent scores that can be expected from beginning eighth-grade 
students. 

Results are presented in the tables below. Exhibits E-l, E-2, and E-3 present information on each 
of the major tests that were used. As can be seen looking across rows, the two forms (pre-test and 
post-test) were roughly equivalent across the tests; the largest difference is on the writing test. 
Also, comparing results for seventh- and eighth-grade students, eighth-grade students 
consistently performed better than seventh-grade students. Again, the writing test showed the 
weakest result and may require some additional modification. A large number of students left the 
writing test blank; in focus groups, students reported that they were unfamiliar with the test 
format or had little or no experience writing English or both. 



Exhibit E-1. IPT 2004 Reading Test Average Raw Score (Total Items = 53) 



Grade 


Form 1C 
(n = 940) 


Form ID 
(n = 907) 


Difference between 
Form 1C and ID 


7th grade 


20.07 


20.58 


-.51 


(n = 1,017) 


SD = 6.08 


SD = 6.68 


t=-1 .27 




(n = 513) 


(n = 504) 


not significant (p>.05) 


8th grade 


29.71 


30.41 


.70 


(n = 830) 


SD = 9.91 


SD = 9.14 


t=1 .06 




(n = 427) 


(n = 403) 


not significant (p>.05) 


Difference between 7th and 


-9.64 


-9.83 




8th grade 


t=-1 8.28 


t=-1 8.07 






p<.05 


p<.05 




Average Total 


24.45 


24.95 




(N = 1,847) 


SD = 9.37 


SD = 9.25 
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Exhibit E-2. MAC II Listening Test Average Raw Score (Total Items = 24) 



Grade 


Form A2 


Form B2 


Difference between Form 




(n = 940) 


(n = 899) 


A2 and Form B2 




12.80 


11.76 


1.04 


7th grade 


SD = 2.96 


SD = 3.08 


t=5.49 


(n = 1018) 


(n = 513) 


(n = 505) 


p<.05 




14.43 


13.60 


.83 


8th grade 


SD = 3.62 


SD = 3.36 


t=3.4 


(n = 821) 


(n = 427) 


(n = 394) 


p<.05 


Difference 


-1.63 


-1.84 




between 7th 


t =-7.59 


t=-8.54 




and 8th grade 


p<.05 


p<.05 




Average Total 


13.54 


12.57 




(N = 1,839) 


SD = 3.37 


SD = 3.33 





Exhibit E-3. IPT 2004 Writing Test Average Raw Score 

(Total Items = 3; Highest Possible Score = 9) 



Grade 


Form 1C 
(n = 922) 


Form ID 
(n = 894) 


7th grade 


1.80 


2.20 


(n = 996; 354 received a 


SD = 1.94 


SD = 2.37 


score of zero) 


(n = 498; 1 89 received a score of 
zero) 


(n = 498; 165 received a score of 
zero) 


8th grade 


2.44 


3.84 


(n = 820; 202 received a 


SD = 2.68 


SD = 2.40 


score of zero) 


(n = 424; 1 54 received a score of 
zero) 


(n = 396; 48 received a score of 
zero) 


Average Total 


2.09 


2.93 


(N = 1,816) 


SD = 2.33 


SD = 2.52 



Exhibit E-4 presents information on the combined score from the reading and listening tests. 
Again, the tests are able to quantify a difference between seventh- and eighth-grade students. 



Exhibit E-4. Reading and Listening Tests Combined Average Raw Score 
(Total Items = 77) 



Grade/Form 


Regular Form 
(n = 932) 


Alternate Form 
(n = 873) 


7th grade 


32.88 


32.40 


(n = 1,002) 


SD = 7.81 


SD = 8.62 




(n = 509) 


(n = 493) 


8th grade 


44.09 


43.99 


(n = 803) 


SD = 12.43 


SD = 11.32 




(n = 423) 


(n = 380) 


Average Total 


37.97 


37.44 


(N = 1,805) 


SD = 11.60 


SD = 11.43 



Exhibit E-5 presents the combined raw score on the reading and writing test by the students’ 
scores on an eighth-grade English test administered by the schools. As expected, students who 
score well on the eighth-grade school test tend to score well on the tests administered by the 
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project, and students who perform poorly on the school tests tend to perform poorly on the tests 
we administered. 



Exhibit E-5. Reading and Listening Combined Raw Score by School Score of 
Eighth-Graders 





Pilot Reading and Listening Combined Score 
Percent Correct 




Score on 8th-Grade 
School Test 
Percent Correct 


76-100% 


51-75% 


26-50% 


0-25% 


Total 


76-100% 


32.2% 


55.3% 


10.5% 


2.0% 


152 


51-75% 


1 1 .6% 


69.5% 


18.1% 


0.8% 


249 


26-50% 


3.8% 


47.3% 


45.8% 


3.1% 


260 


0-25% 


4.2% 


26.0% 


65.6% 


4.2% 


96 


Total 


92 


405 


243 


17 


757 



Finally, the correlation between the subtests appears to be reasonable, as presented in Exhibit 
E-6. The school score is not highly correlated with the listening test. This is consistent with the 
rationale for the study; teachers and students in rural areas have little opportunity to interact with 
native English speakers and consequently their listening (and speaking skills) are frequently 
weaker than their reading skills. 



Exhibit E-6. Correlations Between Tests 





Reading 


Listening 


Writing 


School Score 


Reading 


1 








Listening 


0.56216653 


1 






Writing 


0.64582232 


0.37361175 


1 




School Score 


0.47543256 


0.30729068 


0.498881887 


1 



Finally, RTI may develop one additional short test to be administered at the end of the year. This 
test will be focused on the vocabulary that is unique to TFW and the vocabulary that is shared by 
the school textbook and TFW. 
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Attachment: Schedule of Major Activities 

Schedule of Major Activities for Full-Scale Evaluation 



Activity 


Responsible 


Date 


Identification of sample schools 


PCET 


June 30, 2009 


Random assignment 


RTI 


July 10, 2009 


Invitations to teacher training 


PCET 


July 15, 2009 


LGN/Coastline visit selected schools 


LGN 


July 30 - August 3, 2009 


Teacher training 


LGN/NWN 


August 3-6, 2009 


Final preparation and shipping of 
materials for pre-test 


RTI 


August 15, 2009 


School visits to test equipment and meet 
with teachers 


LGN/NWN 


September 14 - 18, 2009 


Test administrator training 


RTEECNU 


September 28, 2009 


Pre-test administration in schools 


ECNU 


September 29 - October 9, 2009 


Quality control observations 


RTEECNU 


November 23 - 27, 2009 


Final preparation and shipping of 
materials for post-test 


RTI 


April 30, 2010 


Post-test administration 


ECNU 


June, 2010 


Data entry 


ECNU 


June 30, 2010 


Analysis and reporting 


RTI 


July - September 2010 
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RTI Project 0209234.002.008 



September, 2009 

Test Administrator’s Manual 

Evaluation of the E-Learning System: Open- 
Language Learning Initiatives for English 

Main Study 
Lanzhou, Gansu 
China 



Appendix F - Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires 



1. Background and Purpose of the Evaluation 

The U.S. Department of Education [Policy and Program Studies Service (PPSS), within 
the Office of Planning, Evaluation, and Policy Development (OPEPD)], is sponsoring an 
evaluation of the E-language learning system named Open Language Learning Initiative for 
English. (OLLI English). The computer-based game developed to help students learn English is 
called The Forgotten Word (TFW). 

RTI International (RTI) is carrying out the evaluation which is led by Dr. Patricia 
Green. RTI is collaborating with East China Normal University (ECNU) for the work in 
China. The project director at ECNU is Professor Zhu Yiming, a professor of the ECNU 
Department of Education. 

This evaluation has been scheduled to take place in Chinese middle schools for two 
semesters from September 2009 to June 2010. The Chinese Ministry of Education identified 
Gansu Province as a site that fit the requirements for the evaluation. The provincial authorities 
selected the sampled schools. The evaluation aims to answer the following key evaluation 
question: 

Do students exposed to TFW demonstrate increases in proficiency in reading, writing, 
listening, and speaking skills when compared to students without access to TFW, given the same 
curriculum and under conditions that allow reasonable access to the software? 

The role of the test administrators is to: 

1. Confirm (and update if needed) the current student roster with the student tracking form. 

2. Distribute consent forms to all students and read aloud the form. 

3. Administer reading and listening tests. 

4. Administer a questionnaire to all students participating in the survey about their thoughts 
and experience about learning English (attached to the listening test). 

5. Administer a questionnaire to all teachers participating in the teachers’ training for the 
survey. The questionnaire asks about the teachers’ thoughts and experience about teaching 
English. 

2. Verifying the Student Roster with the Student Tracking 
Form 

2.1 Prior to Test Administration - Student Roster vs. Student 

Tracking Form 

The first important step is to obtain a current student roster for each selected classroom at 
the school PRIOR to the test day (or at least 2 hours before testing is scheduled to begin). You 
will need to review each student name listed on the current student roster against the tracking 
form. A sample student tracking form is included in this manual as Attachment 1. 

■ If a student is missing from the student tracking form, you must add this student to the bottom 
of the tracking form on a line that has been assigned an ID but has no name. Please put the 
student’s name on the list. It is very important that we know who has been added to which 
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classroom and school. Please check with these students and make sure they include their 
TFW User ID on the questionnaire, or make a note that they do not have one assigned. 

■ If you notice a spelling change, please update the student tracking form with the correct 
spelling. 

■ If you have more than 3 additional students on the student list in the class NOT listed on the 
tracking form, you should assign new student IDs, using the same school and classroom IDs, 
but adding student IDs beginning with 501. 

2.2 On Test Day 

All test materials have been bundled by classroom. Test materials for each student will be 
inside individual envelopes. The outside of each envelope will list the student’s name and ID - 
YOU MUST DISTRIBUTE THE ENVELOPES TO STUDENTS INDIVIDUALLY, 
MATCHING THE NAME ON THE OUTSIDE OF THE ENVELOPE TO THE CORRECT 
STUDENT. If you have added students to the student tracking form, you must give the newly 
added student the corresponding student packet ID. 

The students should confirm that they have received the correct envelope by verifying the 
name on the outside of the envelope. 

During the test, or after the test, please use the student tracking form to check off which 
students take the test. Please make a check after the student ID on the tracking form. 

3. Consent Forms 

3.1 Parent Letter 

We will give the schools a letter addressed to parents of the 8 th grade students who are 
participating in the study, and ask the schools to distribute the letters to the parents. The letter 
describes the purpose of the study, students’ participation, and confidentiality. Parents who do 
not wish to have their child participate in the study can tear off the bottom portion of the letter 
that says, “I do not wish to have my child participate in this research” and return it to the test 
administrator. This student shall then be excluded from the study. Samples of the letter are 
included in English as Attachments 4 and 5 (and 4a and 5a in Chinese). 

3.2 Student Assent Form 

Before the start of the English language assessment tests and student questionnaire, the 
test administrator should give each student a copy of the student assent form and read it aloud to 
the students. The form will explain the purpose of the study, the voluntary nature of their 
participation, and that they are asked to complete assessment tests for reading and listening, as 
well as a questionnaire about their thoughts and experience with learning English. The form also 
assures the students that their individual test scores or responses to the questionnaire will not be 
shared with their teachers, their parents, China’s Ministry of Education, or the U.S. Department 
of Education. To ensure confidentiality, the students’ name will not be recorded on the tests but 
will be represented by an ID number. If a student does not wish to participate, he or she shall be 
excluded from the study. Copies of the forms are provided in English as Attachments 2 and 3 
(and 2a and 3a in Chinese). 
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3.3 Teacher Consent Form for Participation 

Before administering the teacher’s questionnaire, the administrator should give a consent 
form to each teacher. The form will explain the purpose of the study, the voluntary nature of 
their participation, and that they are asked to complete a questionnaire about their thoughts and 
experience surrounding teaching English. The form also assures the teachers that individual 
responses to the questionnaire will not be shared with the school, other teachers, China’s 
Ministry of Education, or the U.S. Department of Education. To ensure confidentiality, their 
name will not be recorded on the questionnaire but will be represented by an ID number. If a 
teacher does not wish to participate, he or she shall be excluded from the study. Copies of the 
forms are provided in English as Attachments 6 and 7 (and 6a and 7a-Chinese). A copy of the 
Teacher Questionnaire is provided as Attachment 8 and 8a. 

4. Assessment Tests and Questionnaires 

As mentioned previously, you will be administering the reading test, the listening test and 
the student questionnaire to all students sampled for this study. It is very important that you 
follow the scripts for each test - each student must hear the same information and you must 
read/play the entire script for each test. 

4.1 Reading Test 

The reading test is also called the ‘IPT’ test and all items are in one booklet. Test 
administrators distribute the envelopes (that contain both tests and questionnaires) before 
beginning to read the directions for the reading test. After the envelopes are distributed, instruct 
students to take out the reading test booklet. 

When all students have the reading test on their desk, instruct them to open the booklet 
and begin reading the script with the instructions for the first section. 

The reading test has five sections and you will need to provide instructions (from the 
script) before each section. 

A copy of the script (in Chinese and English) is provided in Attachment 9. This script 
must be practiced in advance of administering the test. 

A copy of the reading test is provided as Appendix A. 

4.2 Listening Test 

Directions for the listening test must be played for the students using a cassette tape 
recorder. Before beginning the tape, ask students to take the listening test from the envelope and 
instruct them to open the booklet to the first page. Test administrators should explain to students 
how the bubbles should be marked and then play the standardized recording for the test. Make 
sure that all students understand what to do before the recording is played. As part of the test 
instructions, the recording will cover the function of the bubbles again. The test instructions can 
be replayed, but the actual English prompts should only be played once. 
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A copy of the booklet containing the listening test and questionnaire is in Appendix B. 

4.3 Student Questionnaire 

The student questionnaire is the same booklet as the listening test. This part of the 
assessment asks the student how they feel about learning English. Some students may be getting 
tired of the test at this point, so you should encourage students to complete it and do their best. A 
copy of the student questionnaire is provided in Attachment 9. 

4.4 Teacher Questionnaire 

The teacher questionnaires will be distributed to eighth-grade English teachers on arrival 
to school and collected before departing from the school. A copy of the teacher questionnaire is 
provided in Attachment 8. 

5. Collecting the Test Booklets 

When students have completed the test and questionnaires, collect all booklets from 
students. The tests SHOULD NOT be put back into the envelopes. Please leave the envelopes in 
the classroom or in a trash container at the school. 

Thank the students and teacher for their participation before leaving the classroom. 

Before leaving the school, please check off each booklet, including the teacher 
questionnaire, on the Student Tracking Form. 

Please bundle books from each class together and place in mailing box (do not use boxes 
that were sent from U.S. as they may be damaged). 

DO NOT include the student tracking form in the box with the tests. If for any reason a 
box would be lost or stolen, it is important that a person looking at the tests would be unable to 
link a test with a student name. Please give the tracking forms to Dr. Zhu or Liu Lu. 



Thank you for your time and effort in this important China and U.S. evaluation! 

6. CHECKLIST 

Before going to school, check that you have: 

• Copy of the Student Tracking Form for each class that you will be testing. 

• Envelopes with booklets for all students on Tracking Form(s). 

• Copies of Student Assent Forms. 

• Script with instructions for reading test. 

• Tape and tape recorder for listening test. 
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In the classroom: 

1. Introduce yourself to the class and explain purpose of the study 

2. Give the consent forms to all students. 

3. Distribute the envelopes with test booklets by calling student names. 

4. Read aloud the consent form. 

5. Instruct students to take the reading booklet from the envelope and read instructions for 
the first section of the test. When students have finished first section, read instructions for 
next section. There are 5 sections in the test. 

6. Instruct students to take the listening test booklet from the envelope. Give instructions for 
filling out “bubbles” and begin playing tape recorded instructions. 

7. When students have finished listening test, tell them to turn the page and complete the 
questionnaire. 

8. When students have finished, collect all booklets. Leave envelopes at school. 



After the test session: 

• Complete the Student Tracking Form by checking off the IDs of all completed 
booklets. 

• If any unusual circumstances arose during the test, attach an explanatory note to 
the Student Tracking Form. 
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Attachment 1. Student Tracking Form 



School in English 


School in Chinese 


School ID 


Class# 


Serial Number 


Student Name 


School test ID 


Class test ID 


Student test ID 






10 


1 


1 




910 


201 


301 








1 


2 




910 


201 


302 








1 


3 




910 


201 


303 








1 


4 




910 


201 


304 








1 


5 




910 


201 


305 








1 


6 




910 


201 


306 








1 


7 




910 


201 


307 








1 


8 




910 


201 


308 








1 


9 




910 


201 


309 








1 


10 




910 


201 


310 








1 


11 




910 


201 


311 








1 


12 




910 


201 


312 








1 


13 




910 


201 


313 








1 


14 




910 


201 


314 








1 


15 




910 


201 


315 








1 


16 




910 


201 


316 








1 


17 




910 


201 


317 








1 


18 




910 


201 


318 








1 


19 




910 


201 


319 








1 


20 




910 


201 


320 








1 


21 




910 


201 


321 








1 


22 




910 


201 


322 








1 


23 




910 


201 


323 








1 


24 




910 


201 


324 








1 


25 




910 


201 


325 








1 


26 




910 


201 


326 








1 


27 




910 


201 


327 








1 


28 




910 


201 


328 
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Dear Student: 

Thank you for volunteering to participate in this study. This study is an Evaluation of the Open-Language Learning 
Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a collaboration 
project with China’s Ministry of Education. RTI International, a US-based non-profit research organization, is 
conducting this study. We have also asked East China Normal University (ECNU) to help us. 

The purpose of the study is to examine the effectiveness of English language learning software on students’ English 
proficiency. We are interested in how students listen and read in English. To do this, we will give the students some 
tests that will take about 1.5 hours to complete. In addition, we will ask you to fill out a questionnaire about your 
experience and thoughts with learning English. You are one of about 6,000 middle school students participating in 
this study. 

Your participation is voluntary. You may change your mind and not participate at any point. If you decide not to 
participate, there is no penalty. You may choose not to answer any question. There are no specific risks to you from 
taking part in the study. 

There are no other direct benefits to you from participating, but you will be helping us understand the effectiveness 
of English language learning software on students’ English proficiency. Your name will not be recorded on the tests 
or the questionnaire and you are only represented by an ID. We are required to keep all information confidential and 
what you say or do on these tests will not affect your grades at school. Only authorized researchers at RTI and 
ECNU will know your test scores and your responses to the questionnaire. No one else will know, not your school, 
your teachers, the U.S. Department of Education, China’s Ministry of Education, or your parents. 

If you have any questions about this study or your participation, please feel free to ask the test administrator. 

We thank you in advance for your help! 

Patricia J. Green 
Project Director 
RTI International 
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Attachment 3. Assent Form - Comparison Students - no TFW 

Dear Student: 

Thank you for volunteering to participate in this study. This is a study of the Evaluation of the Open-Language 
Learning Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a 
collaboration project with China’s Ministry of Education. RTT International, a US-based non-profit research 
organization, is conducting this study. We have also asked East China Normal University (ECNU) to help us. 

The purpose of the study is to examine the effectiveness of English language learning. We are interested in how 
students listen and read in English. To do this, we will give the students some tests that will take about 1 .5 hours to 
complete. In addition, we will ask you to fill out a questionnaire about your experience and thoughts with learning 
English. You are one of about 6,000 middle school students participating in this study. 

Your participation is voluntary. You may change your mind and not participate at any point. If you decide not to 
participate, there is no penalty. You may choose not to answer any question. There are no specific risks to you from 
taking part in the study. There are no other direct benefits to you from participating, but you will be helping us 
understand the effectiveness of English language learning. Your name will not be recorded on the tests or the 
questionnaire and you are only represented by an ID. We are required to keep all information confidential and what 
you say or do on these tests will not affect your grades at school. Only authorized researchers at RTI and ECNU will 
know your test scores and your responses to the questionnaire. No one else will know, not your school, your 
teachers, the U.S. Department ofEducation, China’s Ministry of Education, or your parents. 

If you have any questions about this study or your participation, please feel free to ask the test administrator. 

We thank you in advance for your help! 

Patricia J. Green 
Project Director 
RTI International 
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Attachment 4. Parent Letter - Treatment Schools -TFW Parents 

Dear Parent or Guardian: 

We are pleased to inform you that your child’s school has been selected to participate in an important study called 
the Evaluation of the Open-Language Learning Initiative for English (OLLI English). It is sponsored by the U.S. 
Department of Education, as part of a collaboration project with China’s Ministry of Education. RTI International, a 
US-based non-profit research organization, is conducting this study. We have also asked East China Normal 
University (ECNU) to help us. 

The purpose of the study is to examine the effectiveness of English language learning software on students’ English 
proficiency. We would like to measure students’ English language proficiency in listening and reading. To do this, 
we will give the students some tests that will take about 1 .5 hours to complete. In addition, we will ask your child to 
fill out a questionnaire about his or her experience and thoughts with learning English. Your child will be one of 
about 6,000 middle school students participating in this study. Your child’s participation is voluntary. You or your 
child may withdraw from the study at any point. 

There is no penalty if you or your child decides not to participate. Your child may choose not to answer any 
question. There are no specific risks to your child from taking part in the study but he or she might feel slightly 
uncomfortable with the test administrator because your child might not have met him or her before. There are no 
other direct benefits to your child from participating, but he or she will be helping us understand the effectiveness of 
English language learning software on students’ English proficiency. Your child’s name will not be recorded on the 
tests or the questionnaire but will only be represented by an ID. We are required to keep all information confidential 
and your child’s performance on these tests will not affect his or her grades at school. Only authorized researchers at 
RTI and ECNU will know your child’s test scores and responses to the questionnaire. They will not be shared with 
the school, the teachers, the U.S. Department of Education, China’s Ministry of Education, or with you. 

If you do not wish to have your child participate, please tear off the bottom portion of this letter and have your child 
give it to the test administrator. If you have any questions about this study or your child’s participation, please feel 
free to ask the test administrator that will be at your child’s school. We thank you in advance for your cooperation in 
this important study. 

Sincerely, 

Patricia J. Green 
Project Director 
RTI International 



I do not wish to have my child participate in this research. 
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Attachment 5. Parent Letter - Comparison Schools - Non TFW Parents 

Dear Parent or Guardian: 

We are pleased to inform you that your child’s school has been selected to participate in an important study called 
the Evaluation of the Open-Language Learning Initiative for English (OLLI English). It is sponsored by the U.S. 
Department of Education, as part of a collaboration project with China’s Ministry of Education. RTI International, a 
US-based non-profit research organization, is conducting this study. We have also asked East China Normal 
University (ECNU) to help us. 

The purpose of the study is to examine the effectiveness of English language learning. We would like to measure 
students’ English language proficiency in listening and reading. To do this, we will give the students some tests that 
will take about 1 .5 hours to complete. In addition, we will ask your child to fill out a questionnaire about his or her 
experience and thoughts with learning English. Your child will be one of about 6,000 middle school students 
participating in this study. Your child’s participation is voluntary. You or your child may withdraw from the study at 
any point. 

There is no penalty if you or your child decides not to participate. Your child may choose not to answer any 
question. There are no specific risks to your child from taking part in the study. There are no other direct benefits to 
your child from participating, but he or she will be helping us understand the effectiveness of English language 
learning in China. Your child’s name will not be recorded on the tests or the questionnaire but will only be 
represented by an ID. We are required to keep all information confidential and your child’s performance on these 
tests will not affect his or her grades at school. Only authorized researchers at RTI and ECNU will know your 
child’s test scores and responses to the questionnaire. They will not be shared with the school, the teachers, the U.S. 
Department of Education, China’s Ministry of Education, or with you. 

If you do not wish to have your child participate, please tear off the bottom portion of this letter and have your child 
give it to the test administrator. If you have any questions about this study or your child’s participation, please feel 
free to ask the test administrator that will be at your child’s school. We thank you in advance for your cooperation in 
this important study. 

Sincerely, 

Patricia J. Green 
Project Director 
RTI International 



I do not wish to have my child participate in this research. 
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Attachment 6. Consent Form for Participation in the Teacher Questionnaire - TFW 

Dear Teacher: 

Thank you for volunteering to participate in this study. This study is an Evaluation of the Open-Language Learning 
Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a collaboration 
project with China’s Ministry of Education. RTI International, a US-based non-profit research organization, is 
conducting this study. We have also asked East China Normal University (ECNU) to help us. 

The purpose of the study is to examine the effectiveness of English-language learning software on students' English 
proficiency. We would like to measure students’ English language proficiency in listening and reading. At this time, 
we are asking you to fill out a questionnaire about your experience and thoughts with teaching English. It will take 
about 10 minutes to complete. You will be one of about 90 middle school teachers participating in this study. 

Your participation is voluntary. You may withdraw from the study at any point. There is no penalty if you decide 
not to participate. You may choose not to answer any question. There are no specific risks to you from taking part in 
the study. There are no other direct benefits to you from participating, but you will be helping us understand 
teachers’ experiences in teaching English . Your name will not be recorded on the questionnaire, but will be 
represented by an ID. We are required to keep all information confidential and your responses in the questionnaire 
will not affect your status at school. Only authorized researchers at RTI and ECNU will know your responses to the 
questionnaire. They will not be shared with the school, others teachers, the U.S. Department of Education, China’s 
Ministry of Education, or with your students. 

If you have any questions about this study or your participation, please feel free to ask the questionnaire 
administrator. 

We thank you in advance for your help! 

Sincerely, 

Patricia J. Green 
Project Director 
RTI International 
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Attachment 7. Consent Form for Participation in the Teacher Questionnaire - NO 
TFW 

Dear Teacher: 

Thank you for volunteering to participate in this study. This study is an Evaluation of the Open-Language Learning 
Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a collaboration 
project with China’s Ministry of Education. RTI International, a US-based non-profit research organization, is 
conducting this study. We have also asked East China Normal University (ECNU) to help us. 

The purpose of the study is to examine the effectiveness of English-language learning. We would like to measure 
students’ English language proficiency in listening and reading. At this time, we are asking you to fill out a 
questionnaire about your experience and thoughts with teaching English. It will take about 10 minutes to complete. 
You will be one of about 90 middle school teachers participating in this study. 

Your participation is voluntary. You may withdraw from the study at any point. There is no penalty if you decide 
not to participate. You may choose not to answer any question. There are no specific risks to you from taking part in 
the study. There are no other direct benefits to you from participating, but you will be helping us understand 
experiences and thoughts about teaching English. Your name will not be recorded on the questionnaire, but will be 
represented by an ID. We are required to keep all information confidential and your responses in the questionnaire 
will not affect your status at school. Only authorized researchers at RTI and ECNU will know your responses to the 
questionnaire. They will not be shared with the school, others teachers, the U.S. Department of Education, China’s 
Ministry of Education, or with your students. 

If you have any questions about this study or your participation, please feel free to ask the questionnaire 
administrator. 

We thank you in advance for your help! 

Sincerely, 

Patricia J. Green 
Project Director 
RTI International 
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Atachment 2a. Assent Form- Student TFW 



m^^Open-Language Learning Initiative for English (OLLI 

English) 

£IiRRTI InternationalStT^SSo — & cHEo 

in#i7«^n6 



m^pfpi^s^o 






RTI International^^!! i if 

— -J* 

Patricia Green 
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Attachment 3a. Assent Form- Comparison Students - No TFW 

m^^Open-Language Learning Initiative for English (OLLI 

English) 

£IiRRTI InternationalStT^SSo — & cHEo 



, # i± » ;i 9^ Fffift ^ s& « - W , nt 

s-SfiMA 

mm, 

zkfomm ip!t»j^*^ip1o 

£lttAiltiltM#^o 



RTI International^^!! i if 

J. 

Patricia Green 
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Attachment 4a. Parent Letter - TFW Parents 



mmrnwmwt , mmmmms , miopen- 

Language Learning Initiative for English (OLLI 
English) 0 

RTI Internationaffi^T^SSo Ife-fn-tllit — SBpqffo 



9&tf wiriSo rw , 

mimm , im 

ssma , mi 



, wa%mm 



RTI International^^!! i if 

— 'J- / /3' ULtA 

Patricia Green lft± 






o 
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Attachment 5a. Parent Letter - Non TFW Parents 



mmmwmwt , mmmmms , miopen- 

Language Learning Initiative for English (OLLI 
English) 0 

RTI InternationaffitT^SSo ifeCHfeii — SBpqffo 

it warn 

afe^wiPiSo 



smua , mm * t 

SB, *S#W3^&3 : F£ttftSno 






RTI International^^!! i if 

— 'J' / /3' ULtA 

Patricia Green lfe± 






o 
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Attachment 6a. Consent Form for Participation in the Teacher Questionnaire - TFW 

: 

m^^Open-Language Learning Initiative for English (OLLI 

English)M^-35Wteo , MSfeHH 

&5#EK$J£IiRRTI Intemationalffl^T^SSo ^3 ^#0 

Rpf. immjjo fpia 

S-sma , UHlit 

RTI International^^ Jj i ±fr 

■ y^ t - t ^ A/v - 

Patricia Greenlfc± 
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Attachment 7a. Consent Form for Participation in the Teacher Questionnaire - NO 
TFW 

mmzm ■. 

m^^Open-Language Learning Initiative for English (OLLI 

English^JHi&^-Wteo , MSftHH 

&5#EK$]£IiRRTI Internationalffl^T^JSo ^3 ^#0 

urn* 

WSP, 

RTI International^^!! i ±fr 

J- /Km*'- 

Patricia Greenlfc± 
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Attachment 8. Teacher Questionnaire 



The following questions are about your experience and thoughts. There are no right or 
wrong answers. Please read all questions thoroughly and provide a response to the best of 
your ability. Directions are provided at each question. Please mark the box next to your 
answer choice, like this: |V_| For some questions, you will be asked to write your responses 
using words or numbers. 



1. Are you male or female? 



EH Male EH Female 



2. What year were you born? I_1J_9J I I 

3. How many years have you taught English? I I I years 



4. The following questions are about your English classes this semester . 

a. How many classroom hours of English classes do you teach on a weekly basis? 

I I I 

b. How many English classes do you teach? I I classes 

c. What are the grade levels? □ 7 th grade □ 8 th grade □ 9 th grade 



5. Do you have a degree in English? EH No EH Yes 

6. The following questions are about the highest degree you have completed . Using the list on the 
bottom, please fill in the corresponding number. 

a. What is the highest level of degree you have completed? I I 

b. What is the highest level of degree in English you have completed? I I (Write 

0 if you do not have a degree in English) 

List of Highest Degree Completed 

(1) Less than High School 

(2) High School Diploma 

(3) 3- Year Normal High School Diploma 

(4) Associate Degree, including 5 -Year Normal College 

(5) 4- Year University 

(6) Advanced Degree 

7. During this semester , have you participated in teachers’ training related to English teaching 
that was organized by your school? (Mark one answer) 

□ No 

□ Yes -> How many days in total? I I I days 
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8. During this semester , have you participated in teachers’ training related to English teaching 
that was organized by an entity other than your school? (Mark one answer) 

□ No 

□ Yes -> How many days in total? I I I days 

9. Currently, how experienced are you with using instructional technology to improve classroom 
instruction for English? (Mark one answer) 

□ Not experienced at all 
I I Less experienced 

I I Somewhat experienced 
I I Very experienced 

10. Have you used computer software for teaching 8 th grade English classes? (Mark one answer) 

□ No 

□ Yes -> What kind of software? Please specify: 

11. What are some teaching methods that you frequently use for teaching 8 th grade English 
classes? (Mark all that apply) 

I I Give lectures according to textbook chapters 
I | Incorporate supporting materials for classroom instruction 
I I Design English language games and activities 
I | Focus on test-taking skills 
I I Something else -> What are they? Please specify: 



12. In general, what percentage of time do you use Chinese (including lecture, oral questioning, 
dialogues, etc.) to teach 8th grade English classes? 

□ Less than 20% 

□ 20% - 50% 

□ 50% - 80% 

I | More than 80% 

13. Do you have a computer at home? 

Yes 

No 

14. Do you have access to the internet at home? 

Yes 

No 
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15. What is your username for the Forgotten World? 

16. Being as objective as you can, how motivated are your 8 th grade students in terms of learning 
English? 

I | Not motivated at all 
I I Less motivated 
I I Somewhat motivated 
I | Very motivated 

17. Currently, which of the following apply to you? (Mark all that apply) 

I 1 1 have traveled to English-speaking countries 

□ I use English to speak with relatives, family, or friends 

I 1 1 read English language books and magazines, newspapers, and Web sites 

18. How often do you watch English language TV programs for at least 15 minutes in a sitting? 
(Mark one answer) 

I | Never 

I | Once every few months 
I | Once or twice a month 
I I Once or twice a week 
I | Three or four times a week 
I I Everyday 

19. How often do you use English to have online chat with people or write e-mails? 

I | Never 

I | Once every few months 
I | Once or twice a month 
I I Once or twice a week 
I | Three or four times a week 
I I Everyday 

20. How important do you feel that learning English is to middle school students in this area? 
(Mark one answer) 

[31 Not important at all 
I | Less important 
I | Somewhat important 
I I Very important 



Thank you for completing this questionnaire! 
We will keep your responses confidential. 
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Attachment 8a. Teacher Questionnaire (Chinese) 
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Attachment 9. Student Questionnaire 

The following questions are about your experience and thoughts in learning English. There 
are no right or wrong answers. Please read all questions thoroughly and provide a response 
to the best of your ability. Directions are provided at each question. Please mark the box 
next to your answer choice, like this: |VJ For some questions, you will be asked to write 
your response. 

1. Are you a boy or a girl? O Boy □ Girl 

2. What year were you born? I_1 _I_9J_9_I I 

3. In what grade did you first start learning English at school? I I grade 

4. Currently, how important is learning English to you? (Mark one answer) 

I I Not important at all 
I I Not too important 
I I Somewhat important 
I I Very important 

5. Currently, which of the following apply to you? (Mark all that apply) 

I I I have traveled to English-speaking countries 

I I I would use English to speak with relatives, family, or friends 

I I 1 would read English language books and magazines, newspapers, and Web sites 

6. How often do you watch English language TV programs for at least 15 minutes in a 
(Mark one answer) 

I I Never 

I I Once every few months 
I I Once or twice a month 
I I Once or twice a week 
I I Three or four times a week 
I I Everyday 

7. How often do you use English to have online chat with people or write e-mails? (Mark one 
answer) 

I I Never 

I I Once every few months 
I I Once or twice a month 
I I Once or twice a week 
I I Three or four times a week 
I I Everyday 
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8. How often do you play computer games that have vocabulary in English ? 
(Mark one answer) 

I I Never 

I I Once every few months 
I I Once or twice a month 
I I Once or twice a week 
I I Three or four times a week 
I I Everyday 

9. Do you go to an English tutoring class outside of school? (Mark one answer) 

I I Never 

I I 1 have, but not anymore 
I I I do, but have not before 
□ 1 always have 

10. Currently, how interested are you in learning English? (Mark one answer) 

I I Not interested 
I I Less interested 
I I More interested 
I I Very interested 



11. Students in general may experience difficulty when they are learning English. Currently, how 
difficult are the following areas to you? (Mark one answer per row) 



Content 


Level of ] 


Difficulty 


Not Difficult At 
All 


A Little Difficult 


Somewhat 

Difficult 


Very Difficult 


a. 


Vocabulary 


□ 


□ 


□ 


□ 


b. 


Grammar 


u 


□ 


u 


□ 


c. 


Listening 


□ 


□ 


□ 


□ 


d. 


Reading 


u 


□ 


u 


u 


e. 


Writing 


□ 


□ 


□ 


□ 


f. 


Speaking 


□ 


□ 


□ 


□ 



12. Realistically, do you expect to attend university some day? (Mark one answer) 

□ No 
I I Yes 
I I Unsure 



A Study of a Gaming Approach to English Language Learning for Middle School Students 



F-31 





Appendix F - Data Collection Procedures and Materials: 2009 Pre-Test Test Administrator Manual, Attached 
Informed Consent Forms, and Questionnaires 

13. Do you have a computer at home? 

I I Yes 

□ No 

14. Do you have access to the internet at home? 

I I Yes 

□ No 

15. How often do you play computer or video games? 

I I Never 

I I Once every few months 
I I Once or twice a month 
I I Once or twice a week 
I I Three or four times a week 
I I Everyday 

16. What is your username for the Forgotten World? . 



Thank you for completing this questionnaire! 
We will keep your responses confidential. 
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Attachment 9a. Student Questionnaire 
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1. Background and Purpose of the Evaluation 

The U.S. Department of Education [Policy and Program Studies Service (PPSS), within 
the Office of Planning, Evaluation, and Policy Development (OPEPD)], is sponsoring an 
evaluation of the E-language learning system named Open Language Learning Initiative for 
English. (OLLI English). The computer-based game developed to help students learn English is 
called The Forgotten Word (TFW). 

RTI International (RTI) is carrying out the evaluation which is led by Dr. Patricia 
Green. RTI is collaborating with East China Normal University (ECNU) for the work in 
China. The project director at ECNU is Professor Zhu Yiming, a professor of the ECNU 
Department of Education. 

This evaluation has been scheduled to take place in Chinese middle schools for two 
semesters from September 2009 to June 2010. The Chinese Ministry of Education identified 
Gansu Province as a site that fit the requirements for the evaluation. The provincial authorities 
selected the sampled schools. The evaluation aims to answer the following key evaluation 
question: 

Do students exposed to TFW demonstrate increases in proficiency in reading, writing, 
listening, and speaking skills when compared to students without access to TFW, given the same 
curriculum and under conditions that allow reasonable access to the software? 

The role of the test administrators is to: 

1. Confirm (and update if needed) the current student roster with the student tracking form. 

2. Distribute consent forms to all students and read aloud the form. 

3. Administer reading and listening tests. 

4. Administer a questionnaire to all students participating in the survey about their thoughts 
and experience about learning English (attached to the listening test). 

5. Administer a questionnaire to teachers participating in the program study. The 
questionnaire asks about the teachers’ thoughts and experience about teaching English. There 
are two versions of the questionnaire: one for teachers in the “treatment” schools that are 
using The Forgotten World and one for teachers in “comparison” schools. 

2. Verifying the student roster with the Student Tracking 
Form 

2.1 Prior to Test Administration - Student Roster vs. Student 

Tracking Form 

The first important step is to obtain a current student roster for each selected classroom at 
the school PRIOR to the test day (or at least 2 hours before testing is scheduled to begin). You 
will need to review each student name listed on the current student roster against the tracking 
form. A sample student tracking form is included in this manual as Attachment 1. 
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■ If a student is missing from the student tracking form, you must add this student to the bottom 
of the tracking form on a line that has been assigned an ID but has no name. Please put the 
student’s name on the list. It is very important that we know who has been added to which 
classroom and school. Please check with these students and make sure they include their 
TFW User ID on the questionnaire, or make a note that they do not have one assigned. 

■ If you notice a spelling change, please update the student tracking form with the correct 
spelling. 

■ If you have more than 3 additional students on the student list in the class NOT listed on the 
tracking form, you should assign new student IDs, using the same school and classroom IDs, 
but adding student IDs beginning with 601. 

2.2 On Test Day 

All test materials have been bundled by classroom. Test materials for each student will be 
inside individual envelopes. The outside of each envelope will list the student’s name and ID - 
YOU MUST DISTRIBUTE THE ENVELOPES TO STUDENTS INDIVIDUALLY, 
MATCHING THE NAME ON THE OUTSIDE OF THE ENVELOPE TO THE CORRECT 
STUDENT. If you have added students to the student tracking form, you must give the newly 
added student the corresponding student packet ID. 

The students should confirm that they have received the correct envelope by verifying the 
name on the outside of the envelope. 

During the test, or after the test, please use the student tracking form to check off 
which students take the test. Please make a check after the student ID on the tracking form. 

3. Consent Forms 

3.1 Parent Letter 

We will give the schools a letter addressed to parents of the 8 th grade students who are 
participating in the study, and ask the schools to distribute the letters to the parents. The letter 
describes the purpose of the study, students’ participation, and confidentiality. Parents who do 
not wish to have their child participate in the study can tear off the bottom portion of the letter 
that says, “I do not wish to have my child participate in this research” and return it to the test 
administrator. This student shall then be excluded from the study. Samples of the letter are 
included in English as Attachments 4 and 5 (and 4a and 5a in Chinese). 

3.2 Student Assent Form 

Before the start of the English language assessment tests and student questionnaire, the 
test administrator should give each student a copy of the student assent form and read it aloud to 
the students. The form will explain the purpose of the study, the voluntary nature of their 
participation, and that they are asked to complete assessment tests for reading and listening, as 
well as a questionnaire about their thoughts and experience with learning English. The form also 
assures the students that their individual test scores or responses to the questionnaire will not be 
shared with their teachers, their parents, China’s Ministry of Education, or the U.S. Department 
of Education. To ensure confidentiality, the students’ name will not be recorded on the tests but 



G-6 



A Study of a Gaming Approach to English Language Learning for Middle School Students 



Appendix G - Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires 

will be represented by an ID number. If a student does not wish to participate, he or she shall be 
excluded from the study. Copies of the forms are provided in English as Attachments 2 and 3 
(and 2a and 3a in Chinese). 

3.3 Teacher Consent Form for Participation 

Before administering the teacher’s questionnaire, the administrator should give a consent 
form to each teacher. The form will explain the purpose of the study, the voluntary nature of 
their participation, and that they are asked to complete a questionnaire about their thoughts and 
experience surrounding teaching English. The form also assures the teachers that individual 
responses to the questionnaire will not be shared with the school, other teachers, China’s 
Ministry of Education, or the U.S. Department of Education. If a teacher does not wish to 
participate, he or she shall be excluded from the study. Copies of the forms are provided in 
English as Attachments 6 and 7 (and 6a and 7a-Chinese). A copy of the Teacher Questionnaire 
for treatment schools is provided as Attachment 8 and 8a. The Teacher Questionnaire for 
comparison schools is provided as Attachment 9 and 9a. 

4. Assessment Tests and Questionnaires 

As mentioned previously, you will be administering the reading test, the listening test and 
the student questionnaire to all students sampled for this study. It is very important that you 
follow the scripts for each test - each student must hear the same information and you must 
read/play the entire script for each test. 

4.1 Reading Test 

The reading test is also called the ‘IPT’ test and all items are in one booklet. Test 
administrators distribute the envelopes (that contain both tests and questionnaires) before 
beginning to read the directions for the reading test. After the envelopes are distributed, instruct 
students to take out the reading test booklet. 

When all students have the reading test on their desk, instruct them to open the booklet 
and begin reading the script with the instructions for the first section. 

The reading test has five sections and you will need to provide instructions (from the 
script) before each section. 

A copy of the script (in Chinese and English) is provided in Attachment 10. This script 
must be practiced in advance of administering the test. 

A copy of the reading test is provided as Appendix A. 

4.2 Listening Test 

Directions for the listening test must be played for the students using a cassette tape 
recorder. Before beginning the tape, ask students to take the listening test from the envelope and 
instruct them to open the booklet to the first page. Test administrators should explain to students 
how the bubbles should be marked and then play the standardized recording for the test. Make 
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sure that all students understand what to do before the recording is played. As part of the test 
instructions, the recording will cover the function of the bubbles again. The test instructions can 
be replayed, but the actual English prompts should only be played once. 

A copy of the booklet containing the listening test and questionnaire is in Appendix B. 

4.3 Student Questionnaire 

The student questionnaire is the same booklet as the listening test. This part of the 
assessment asks the student how they feel about learning English. Some students may be getting 
tired of the test at this point, so you should encourage students to complete it and do their best. 

4.4 Teacher Questionnaire 

The teacher questionnaires will be distributed to eighth-grade English teachers on arrival 
to school and collected before departing from the school. A copy of the teacher questionnaire for 
treatment schools is provided in Attachment 8 and for comparison schools in Attachment 9. 

5. Collecting the Test Booklets 

When students have completed the test and questionnaires, collect all booklets from 
students. The tests SHOULD NOT be put back into the envelopes. Please leave the envelopes in 
the classroom or in a trash container at the school. 

Thank the students and teacher for their participation before leaving the classroom. 

Before leaving the school, please check off each booklet, including the teacher 
questionnaire, on the Student Tracking Form. 

Please bundle books from each class together and place in mailing box (do not use boxes 
that were sent from U.S. as they may be damaged). 

DO NOT include the student tracking form in the box with the tests. If for any reason a 
box would be lost or stolen, it is important that a person looking at the tests would be unable to 
link a test with a student name. Please give the tracking forms to Dr. Zhu. 

Thank you for your time and effort in this important China and U.S. evaluation! 

6. Test Procedures 

Before going to school, check that you have: 

■ Copy of the Student Tracking Form for each class that you will be testing. 

■ Envelopes with booklets for all students on Tracking Form(s). 

■ Copies of Student Assent Forms. 

■ Script with instructions for reading test. 

■ Tape and tape recorder for listening test. 

In the classroom: 
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Step 1: Before distribute test booklets to the students 

■ Introduce yourself to the class and explain purpose of the study. 

■ Give the consent forms to all students. 

■ Read aloud the consent form statement to the students. 

Step 2: Distribute test booklets 

■ Distribute the envelopes with test booklets by calling student names one by one. Test 
administrators MUST distribute the tests on their own. Do NOT ask teachers and students to 
help. Tell students that they should NOT open the envelope until everyone is ready. 

■ Instruct students to take the reading booklet from the envelope together. Tell students NOT to 
write down their names on the test booklets. Give a brief introduction on the test booklets. 
Reading test is in a separate booklet. Listening test and student questionnaire are in one test 
booklet together. 

■ Ask students to check if the 9 digit number on the envelope matches with the numbers on the 
two test booklets. If there are any errors, make a note of the ones which don’t match, and 
make changes on the test booklets and rosters accordingly. 

■ Tell students that the whole test takes one and a half hours. The order of the tests is reading 
first (60 minutes), listening second (20 minutes) and student questionnaire last (10 minutes). 

Step 3: Start reading test 

■ Start reading test first. Tell students that they don’t need to answer questions on page 16. 

■ Test administrators must read the instructions by following test administrator manual on page 
35 (attachment 10). The instructions must be read VERBATIM and no sentence should be 
skipped. Test administrator MUST read the instruction on their own. After the first part 
instruction, test administrators need to wait to start the next part till all the students are done 
with first part of the questions. Ask the students who answer the questions faster to wait to 
start the next section. There are 5 sections in the test. 

■ Ask students to cover their answers by using the yellow envelope. Emphasize to the students 
that they must NOT look at other people’s answers. If they don’t know the answer, they may 
leave it blank, but NO copying from other students. Test administrators will need to 
REPETITIVELY remind the students through the whole test. 

■ The time distribution of the reading test is described as follows: 

Part 1: Vocabulary 5-10 minutes 

Part 2: Vocabulary in context 5-10 minutes 
Part 3: Reading for understanding 15-30 minutes 
Part 4: Reading for life skills 8-15 minutes 
Part 5: Language usage 4-9 minutes 

■ When students finish reading test, ask them to put the test booklets on the left corner of the 
table and start the listening test. 

Step 4: Start listening test and student questionnaire 

■ All schools should use tape player or broadcast system for listening test. 

■ If broadcast system is being used in the listening test, all the classes must start the listening 
test TOGETHER after reading test. 

■ Instruct students to take the listening test booklet from the envelope. Give instructions for 
filling out “bubbles” and begin playing tape recorded instructions. 
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■ When students have finished listening test, tell them to turn the page and complete the 
questionnaire. Remind the students to read the instruction on the questionnaire before start 
answering questions. 

■ When students work on the questionnaire, test administrators can rewind the tape for the next 
use. 

Step 5: Collect test booklets 

■ Before you collect the test booklets, tell students they do NOT need to put the finished test 
booklets into the envelope. Tell students to keep the student consent forms and envelope. 

■ Ask the students to put the finished test booklets on the corner of the table. Test 
administrators must collect the test booklets one for one and check the two booklets. After all 
the test booklets are collected, test administrators thank students for participating and ask 
students to dismiss. 

Step 6: After the test session: 

■ Complete the Student Tracking Form by checking off the IDs of all completed booklets. 

■ If any unusual circumstances arose during the test, attach an explanatory note to the Student 
Tracking Form. 
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Attachment 1. Student Tracking Form 



School in English 


School in Chinese 
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22 
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23 
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24 
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25 
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1 


26 
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1 


27 
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1 
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Attachment 2. Assent Form - Student - TFW 

Dear Student: 

Thank you for volunteering to participate in this study. This study is an Evaluation of the Open-Language Learning 
Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a collaboration 
project with China’s Ministry of Education. RTI International, a US-based non-profit research organization, is 
conducting this study. We have also asked East China Normal University (ECNU) to help us. 

The purpose of the study is to examine the effectiveness of English language learning software on students’ English 
proficiency. We are interested in how students listen and read in English. To do this, we will give the students some 
tests that will take about 1.5 hours to complete. In addition, we will ask you to fill out a questionnaire about your 
experience and thoughts with learning English and using The Forgotten World software. You are one of about 6,000 
middle school students participating in this study. 

Your participation is voluntary. You may change your mind and not participate at any point. If you decide not to 
participate, there is no penalty. You may choose not to answer any question. There are no specific risks to you from 
taking part in the study. 

There are no other direct benefits to you from participating, but you will be helping us understand the effectiveness 
of English language learning software on students’ English proficiency. Your name will not be recorded on the tests 
or the questionnaire and you are only represented by an ID. We are required to keep all information confidential and 
what you say or do on these tests will not affect your grades at school. Only authorized researchers at RTI and 
ECNU will know your test scores and your responses to the questionnaire. No one else will know, not your school, 
your teachers, the U.S. Department of Education, China’s Ministry of Education, or your parents. 

If you have any questions about this study or your participation, please feel free to ask the test administrator. 

We thank you in advance for your help! 

Patricia J. Green 
Project Director 
RTI International 
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Attachment 3. Assent Form - Comparison Students - no TFW 

Dear Student: 

Thank you for volunteering to participate in this study. This is a study of the Evaluation of the Open-Language 
Learning Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a 
collaboration project with China’s Ministry of Education. RTI International, a US-based non-profit research 
organization, is conducting this study. We have also asked East China Normal University (ECNU) to help us. 

The purpose of the study is to examine the effectiveness of English language learning. We are interested in how 
students listen and read in English. To do this, we will give the students some tests that will take about 1.5 hours to 
complete. In addition, we will ask you to fill out a questionnaire about your experience and thoughts with learning 
English. You are one of about 6,000 middle school students participating in this study. 

Your participation is voluntary. You may change your mind and not participate at any point. If you decide not to 
participate, there is no penalty. You may choose not to answer any question. There are no specific risks to you from 
taking part in the study. There are no other direct benefits to you from participating, but you will be helping us 
understand the effectiveness of English language learning. Your name will not be recorded on the tests or the 
questionnaire and you are only represented by an ID. We are required to keep all information confidential and what 
you say or do on these tests will not affect your grades at school. Only authorized researchers at RTI and ECNU will 
know your test scores and your responses to the questionnaire. No one else will know, not your school, your 
teachers, the U.S. Department ofEducation, China’s Ministry of Education, or your parents. 

If you have any questions about this study or your participation, please feel free to ask the test administrator. 

We thank you in advance for your help! 

Patricia J. Green 
Project Director 
RTI International 
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Attachment 4. Parent Letter - Treatment Schools -TFW Parents 

Dear Parent or Guardian: 

We are pleased to inform you that your child’s school has been selected to participate in an important study called 
the Evaluation of the Open-Language Learning Initiative for English (OLLI English). It is sponsored by the U.S. 
Department of Education, as part of a collaboration project with China’s Ministry of Education. RTI International, a 
US-based non-profit research organization, is conducting this study. We have also asked East China Normal 
University (ECNU) to help us. 

The purpose of the study is to examine the effectiveness of English language learning software on students’ English 
proficiency. We would like to measure students’ English language proficiency in listening and reading. To do this, 
we will give the students some tests that will take about 1.5 hours to complete. In addition, we will ask your child to 
fill out a questionnaire about his or her experience and thoughts with learning English and using The Forgotten 
World software. Your child will be one of about 6,000 middle school students participating in this study. Your 
child’s participation is voluntary. You or your child may withdraw from the study at any point. 

There is no penalty if you or your child decides not to participate. Your child may choose not to answer any 
question. There are no specific risks to your child from taking part in the study but he or she might feel slightly 
uncomfortable with the test administrator because your child might not have met him or her before. There are no 
other direct benefits to your child from participating, but he or she will be helping us understand the effectiveness of 
English language learning software on students’ English proficiency. Your child’s name will not be recorded on the 
tests or the questionnaire but will only be represented by an ID. We are required to keep all information confidential 
and your child’s performance on these tests will not affect his or her grades at school. Only authorized researchers at 
RTI and ECNU will know your child’s test scores and responses to the questionnaire. They will not be shared with 
the school, the teachers, the U.S. Department of Education, China’s Ministry of Education, or with you. 

If you do not wish to have your child participate, please tear off the bottom portion of this letter and have your child 
give it to the test administrator. If you have any questions about this study or your child’s participation, please feel 
free to ask the test administrator that will be at your child’s school. We thank you in advance for your cooperation in 
this important study. 

Sincerely, 

Patricia J. Green 
Project Director 
RTI International 



I do not wish to have my child participate in this research. 
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Attachment 5. Parent Letter - Comparison Schools - Non TFW Parents 

Dear Parent or Guardian: 

We are pleased to inform you that your child’s school has been selected to participate in an important study called 
the Evaluation of the Open-Language Learning Initiative for English (OLLI English). It is sponsored by the U.S. 
Department of Education, as part of a collaboration project with China’s Ministry of Education. RTI International, a 
US-based non-profit research organization, is conducting this study. We have also asked East China Normal 
University (ECNU) to help us. 

The purpose of the study is to examine the effectiveness of English language learning. We would like to measure 
students’ English language proficiency in listening and reading. To do this, we will give the students some tests that 
will take about 1 .5 hours to complete. In addition, we will ask your child to fill out a questionnaire about his or her 
experience and thoughts with learning English. Your child will be one of about 6,000 middle school students 
participating in this study. Your child’s participation is voluntary. You or your child may withdraw from the study at 
any point. 

There is no penalty if you or your child decides not to participate. Your child may choose not to answer any 
question. There are no specific risks to your child from taking part in the study. There are no other direct benefits to 
your child from participating, but he or she will be helping us understand the effectiveness of English language 
learning in China. Your child’s name will not be recorded on the tests or the questionnaire but will only be 
represented by an ID. We are required to keep all information confidential and your child’s performance on these 
tests will not affect his or her grades at school. Only authorized researchers at RTI and ECNU will know your 
child’s test scores and responses to the questionnaire. They will not be shared with the school, the teachers, the U.S. 
Department of Education, China’s Ministry of Education, or with you. 

If you do not wish to have your child participate, please tear off the bottom portion of this letter and have your child 
give it to the test administrator. If you have any questions about this study or your child’s participation, please feel 
free to ask the test administrator that will be at your child’s school. We thank you in advance for your cooperation in 
this important study. 

Sincerely, 



Patricia J. Green 
Project Director 
RTI International 



I do not wish to have my child participate in this research. 
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Attachment 6. Consent Form for Participation in the Teacher Questionnaire - TFW 

Dear Teacher: 

Thank you for volunteering to participate in this study. This study is an Evaluation of the Open-Language Learning 
Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a collaboration 
project with China’s Ministry of Education. RTI International, a US-based non-profit research organization, is 
conducting this study. We have also asked East China Normal University (ECNU) to help us. 

The purpose of the study is to examine the effectiveness of English-language learning software on students' English 
proficiency. We would like to measure students’ English language proficiency in listening and reading. At this time, 
we are asking you to fill out a questionnaire about your experience and thoughts with teaching English and using 
The Forgotten World software. It will take about 10 minutes to complete. You will be one of about 90 middle school 
teachers participating in this study. 

Your participation is voluntary. You may withdraw from the study at any point. There is no penalty if you decide 
not to participate. You may choose not to answer any question. There are no specific risks to you from taking part in 
the study. There are no other direct benefits to you from participating, but you will be helping us understand 
teachers’ experiences in teaching English. Your name will not be recorded on the questionnaire, but will be 
represented by an ID. We are required to keep all information confidential and your responses in the questionnaire 
will not affect your status at school. Only authorized researchers at RTI and ECNU will know your responses to the 
questionnaire. They will not be shared with the school, others teachers, the U.S. Department of Education, China’s 
Ministry of Education, or with your students. 

If you have any questions about this study or your participation, please feel free to ask the questionnaire 
administrator. 

We thank you in advance for your help! 

Sincerely, 

Patricia J. Green 
Project Director 
RTI International 
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Attachment 7. Consent Form for Participation in the Teacher Questionnaire - NO 
TFW 

Dear Teacher: 

Thank you for volunteering to participate in this study. This study is an Evaluation of the Open-Language Learning 
Initiative for English (OLLI English). It is sponsored by the U.S. Department of Education, as part of a collaboration 
project with China’s Ministry of Education. RTI International, a US-based non-profit research organization, is 
conducting this study. We have also asked East China Normal University (ECNU) to help us. 

The purpose of the study is to examine the effectiveness of English-language learning. We would like to measure 
students’ English language proficiency in listening and reading. At this time, we are asking you to fill out a 
questionnaire about your experience and thoughts with teaching English. It will take about 10 minutes to complete. 
You will be one of about 90 middle school teachers participating in this study. 

Your participation is voluntary. You may withdraw from the study at any point. There is no penalty if you decide 
not to participate. You may choose not to answer any question. There are no specific risks to you from taking part in 
the study. There are no other direct benefits to you from participating, but you will be helping us understand 
experiences and thoughts about teaching English. Your name will not be recorded on the questionnaire, but will be 
represented by an ID. We are required to keep all information confidential and your responses in the questionnaire 
will not affect your status at school. Only authorized researchers at RTI and ECNU will know your responses to the 
questionnaire. They will not be shared with the school, others teachers, the U.S. Department of Education, China’s 
Ministry of Education, or with your students. 

If you have any questions about this study or your participation, please feel free to ask the questionnaire 
administrator. 

We thank you in advance for your help! 

Sincerely, 

Patricia J. Green 
Project Director 
RTI International 
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Attachment 2a. Assent Form- Student TFW 



iitilt# i m^^Open-Language Learning Initiative for English (OLLI 

English) 

£IiRRTI InternationalStT^SSo — & cHEo 

££±iftn/K xmmtio 

<«»&&##> mmmm 

*I>£c 

, pa&fa&m 



m^pfpi^s^o 

f^^^Pilo 






RTI International#^ 1 !! i if 

— -J* 

Patricia Green 
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Attachment 3a. Assent Form- Comparison Students - No TFW 

m^^Open-Language Learning Initiative for English (OLLI 

English) 

£IiRRTI InternationalStT^SSo — & cHEo 



, # i± » ;i 9^ pfu^ ^ s& « - W , nt 

s-SfiMA 

mm, 

zkfomm ip!t»j^*^ip1o 



RTI International^^:® i if 

— -,J ■ 

Patricia Green 
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Attachment 4a. Parent Letter - TFW Parents 



mmmwmwt , mmmmms , miopen- 

Language Learning Initiative for English (OLLI 
English) 0 

RTI InternationaliS^T^SSo ifed-tei# — SBpqffo 

m p?i )£ 0 
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9&tf wiriSo rw , 
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Attachment 5a. Parent Letter - Non TFW Parents 



mmmwmwt , mmmmms , miopen- 

Language Learning Initiative for English (OLLI 
English) 0 

RTI InternationaffitT^SSo tfeiW — ^3-nfEo 
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RTI International^^!! i if 
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Attachment 6a. Consent Form for Participation in the teacher Questionnaire - TFW 

: 

m^^Open-Language Learning Initiative for English (OLLI 

English) 

£IiRRTI InternationalStT^fiSo 0 cHEo 

iffv immtio 
iL — o 

h^4iri#± 0 

S'SMA , UHlit 

RTI International#]^® S ±W 
Patricia Greenlfc± 
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Attachment 7a. Consent Form for Participation in the Teacher Questionnaire - NO 
TFW 

: 

m^^Open-Language Learning Initiative for English (OLLI 

English) 

£SJRRTI Internationaffl^T^SSo ifedtilii & n'fEo 

sia ± 0 

WSIR 

^isrpiif^a^fpio 

RTI International^^!! i ±fr 

J- 

Patricia Greenlfc± 
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Attachment 8. Teacher Questionnaire (Treatment Schools) 

Teacher ID : 

The following questions are about your experience and thoughts. There are no right or wrong 
answers. Please read all questions thoroughly and provide a response to the best of your ability. 
Directions are provided at each question. Please mark the box next to your answer choice, like 
this: v_ For some questions, you will be asked to write your response using words or numbers. 

1. What year were you born? I_l_l_9_l 1 I 

2. We would like to know if the training you received on using The Forgotten World was helpful. 
How helpful was the training to you? (Mark one answer) 

0 Not helpful at all 

1 I Not so helpful 

I I Somewhat helpful 
I I Very helpful 

3. How often did you use The Forgotten World in class ? (Mark one answer) 

I I Never 

I | Once every few months 
I | Once or twice a month 
I I Once or twice a week 
I | Three to four times a week 
I I Everyday 

4. How often did you organize students to use The Forgotten World outside of class at school? 
(Mark one answer) 

I I Never 

I | Once every few months 
I | Once or twice a month 
I I Once or twice a week 
I | Three to four times a week 
I I Everyday 

5. Now that you have used The Forgotten World, realistically, how likely do you think you will 
use it as part of the curriculum in the future? (Mark one answer) 

0 Not likely at all 

1 | Not too likely 

I I Somewhat likely 
I | Very likely 
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6. Could you tell us more about your answer in Question 5? 



7. Realistically, how likely do you think you will use some other computer software as part of 
the curriculum in the future? (Mark one answer) 

0 Not likely at all 

1 I Not too likely 

I I Somewhat likely 
I | Very likely 

8. In general, how helpful do you think The Forgotten World is to students’ English learning? 
(Mark one answer) 

I I Not helpful at all 
I I Not so helpful 
I I Somewhat helpful 
I I Very helpful 



9. In what area do you think that The Forgotten World can be helpful to students’ English 
learning? (Mark one answer per row) 



Area 


Level of H 


[elpfulness 


Not helpful at all 


A little helpful 


Somewhat 

helpful 


Very helpful 


a. 


Vocabulary 


□ 


□ 


□ 




b. 


Sentence 


□ 


□ 


□ 


□ 


c. 


Listening 


u 


u 


u 


□ 


d. 


Reading 


□ 


□ 


□ 


□ 


e. 


Writing 


u 


u 


u 


u 


f. 


Speaking 


□ 


□ 


□ 


□ 


g- 


Test-taking 


u 


□ 


u 


u 



10. Compared to the past, has student’s motivation for learning English changed since using The 
Forgotten World ? (Mark one answer) 

I I No change at all 
I | A little more motivated 
I | Somewhat more motivated 
! | A lot more motivated 
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11. What were the two biggest challenges that you encountered while using The Forgotten World? 
(Mark two answers) 

I | Insufficient time for using the software 
I | Limited support from the school 

I | Problems with computers at the school computer laboratory 
I I Concerns with negative impact on test scores 

I I Other challenges -> Please specify: 

12. In your opinion, what are some important procedures that need to be implemented so that 
The Forgotten World could be used to improve English instruction and students’ English 
proficiency? 



13. For which grade level would The Forgotten World be most helpful to the students? (Mark one 
answer) 

I I 7 th grade 
I I 8 th grade 
I I 9 th grade 

14. How often did you collaborate or communicate with other teachers in using The Forgotten 
World ? 

I | Never 

I | Once every few months 
I | Once or twice a month 
I I Once or twice a week 
I | Three to four times a week 
I I Everyday 

15. Has using The Forgotten World changed the way you think about teaching? 

I I Yes 
□ No 

If yes, please explain.. 

16. Which 8 th grade classes are you teaching? 

17. Please list ALL the TFW usernames you have been using and in which classroom. 

TFW username: , used in classroom 

TFW username: , used in classroom 
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TFW username: , used in classroom 

If you have TFW usernames for other use besides classroom teaching, please specify. 



Thank you for completing this questionnaire ! 
We will keep your responses confidential. 
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Attachment 8a. 

atttteujtimas , *aywwi£tt®#o ifiiiiif , , » 

L1J_9J I I 

2. 7 The Forgotten 

Woridl&ftfemm , M ) 

□ RGW® 

□ 

□ WBb&A 

□ W&Jj'iRA 

3. The Forgotten World ) 

□ 

□ /l/N3ttE i/£ 

□ 1312ft 

□ §JH£E3il4/fc 

4. Forgotten World ? 

( ) 

□ 

□ /l/N3ftE 1& 

□ $EfcE3ll4/fc 



G-28 



A Study of a Gaming Approach to English Language Learning for Middle School Students 




Appendix G - Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires 

5. Forgotten World 

□ 

□ 

6. m &IE£J1I5®* : 



I I Rlib 

□ Rrt£1W* 

□ W^AWtb 

□ nrai 

8. , MiK^The Forgotten 

□ 

□ Iff !&$&'> 

□ lfffl&&* 

□ IffS&fB* 
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9 . The Forgotten 

£is? ( , SfrRUgjfc-^ ) 





mmm 








SISA 


a. 






□ 






b. 


il>£ 


□ 


□ 


□ 


□ 


c. 


a frlJ 


□ 


□ 


□ 


□ 


d. 


mm 


□ 


□ 


□ 


□ 


e. 


WfF 


□ 


□ 


□ 


□ 


f. 


Pip 


□ 


□ 


□ 


□ 


g- 


#i*t 


□ 


□ 


□ 


□ 



10 . The Forgotten World 

») 

□ >£^ib 
I I 

I I Wfe 



I I WfSAJlil) 
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11 . Forgotten World , JSPJ 7 ? 

( ) 

I I : 



12 . Forgotten 

: 



13 . Mik^ The Forgotten 

□ 7^i 

□ 8^i 

□ 9 ¥£$ 



14 . Mfc$.ffiThe Forgotten World£fc?J$fc ‘ ^ ? 



□ M'f'pMxiji 

□ /l/M£ftefcX3iE l/fc 



□ $jg£^X3ft3Sl4/fc 
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15. The Forgotten World^^j EH) ? 

16. ? 

17. t&Pfi%The Forgotten 

tfw 

tfw 

tfw 

tnm*tiThe Forgotten , HIJUttfcB. 



***UJIlH|**|fi]* ! 
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Attachment 9. Teacher Questionnaire (Comparison Schools) 



Teacher ID : 



The following questions are about your experience and thoughts. There are no right or 
wrong answers. Please read all questions thoroughly and provide a response to the best of 
your ability. Directions are provided at each question. Please mark the box next to your 
answer choice, like this: |VJ For some questions, you will be asked to write your responses 
using words or numbers. 



1. Are you male or female? 



EH Male EH Female 



2. What year were you born? I_1_I_9J I I 

3. How many years have you taught English? I I I years 



4. The following questions are about your English classes this semester . 

a. How many classroom hours of English classes do you teach on a weekly basis? 
I I I 

f. How many English classes do you teach? I I classes 

g. What are the grade levels? □ 7 th grade □ 8 th grade □ 9 th grade 



5. Do you have a degree in English? EH No EH Yes 

6. The following questions are about the highest degree you have completed . Using the list on the 
bottom, please fill in the corresponding number. 

a. What is the highest level of degree you have completed? I I 

b. What is the highest level of degree in English you have completed? I I (Write 

0 if you do not have a degree in English) 

List of Highest Degree Completed 

(1) Less than High School 

(2) High School Diploma 

(3) 3- Year Normal High School Diploma 

(4) Associate Degree, including 5-Year Normal College 

(5) 4- Year University 

(6) Advanced Degree 

7. During this semester , have you participated in teachers’ training related to English teaching 
that was organized by your school? (Mark one answer) 

□ No 

□ Yes -> How many days in total? I I I days 
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8. During this semester , have you participated in teachers’ training related to English teaching 
that was organized by an entity other than your school? (Mark one answer) 

□ No 

□ Yes -> How many days in total? I I I days 

9. Currently, how experienced are you with using instructional technology to improve classroom 
instruction for English? (Mark one answer) 

□ Not experienced at all 
I I Less experienced 

I I Somewhat experienced 
I I Very experienced 

10. Have you used computer software for teaching 8 th grade English classes? (Mark one answer) 

□ No 

□ Yes -> What kind of software? Please specify: 

11. What are some teaching methods that you frequently use for teaching 8 th grade English 
classes? (Mark all that apply) 

I | Give lectures according to textbook chapters 
I | Incorporate supporting materials for classroom instruction 
I I Design English language games and activities 
I I Focus on test-taking skills 

I I Something else -> What are they? Please specify: 



12. In general, what percentage of time do you use Chinese (including lecture, oral questioning, 
dialogues, etc.) to teach 8th grade English classes? 

□ Less than 20% 

□ 20% - 50% 

□ 50% - 80% 

I | More than 80% 

13. Do you have a computer at home? 

Yes 

No 

14. Do you have access to the internet at home? 

Yes 

No 
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15. Being as objective as you can, how motivated are your 8 th grade students in terms of learning 
English? 

I | Not motivated at all 
I I Less motivated 
I I Somewhat motivated 
I | Very motivated 

16. Currently, which of the following apply to you? (Mark all that apply) 

I 1 1 have traveled to English-speaking countries 

□ I use English to speak with relatives, family, or friends 

I 1 1 read English language books and magazines, newspapers, and Web sites 

17. How often do you watch English language TV programs for at least 15 minutes in a sitting? 
(Mark one answer) 

I | Never 

I | Once every few months 
I | Once or twice a month 
I I Once or twice a week 
I | Three or four times a week 
I I Everyday 

18. How often do you use English to have online chat with people or write e-mails? 

I | Never 

I | Once every few months 
I | Once or twice a month 
I I Once or twice a week 
I | Three or four times a week 
I I Everyday 

19. How important do you feel that learning English is to middle school students in this area? 
(Mark one answer) 

Q Not important at all 
I | Less important 
I I Somewhat important 
I I Very important 

Thank you for completing this questionnaire! 

We will keep your responses confidential. 
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Attachment 10. Student Questionnaire 

The following questions are about your experience and thoughts. There are no right or wrong 
answers. Please read all questions thoroughly and provide a response to the best of your ability. 
Directions are provided at each question. Please mark the box next to your answer choice, like 
this: v_ For some questions, you will be asked to write your response. 

1. Are you a boy or a girl? O Boy O Girl 

2. Currently, how important is learning English to you? (Mark one answer) 

O Not important at all 

0 Not too important 

1 | Somewhat important 
I | Very important 

3. Currently, how interested are you in learning English? (Mark one answer) 

0 Not interested 

1 I Less interested 

I | More interested 
I I Very interested 

4. Currently, which of the following apply to you? (Mark all that apply) 

I 1 1 have traveled to English-speaking countries 

I 1 1 would use English to speak with relatives, family, or friends 

I 1 1 would read English language books and magazines, newspapers, and Web sites 

5. How often do you watch English language TV programs for at least 15 minutes in a sitting? 
(Mark one answer) 

I | Never 

I | Once every few months 
I | Once or twice a month 
I I Once or twice a week 
I | Three or four times a week 
I I Everyday 

6. How often do you use English to have online chat with people or write e-mails? (Mark one 
answer) 

I I Never 

I | Once every few months 
I | Once or twice a month 
I I Once or twice a week 
I | Three or four times a week 



G-40 



A Study of a Gaming Approach to English Language Learning for Middle School Students 




Appendix G - Data Collection Procedures and Materials: 2010 Post-Test Test Administrator Manual, Attached 

Informed Consent Forms, and Questionnaires 



I I Everyday 

7. How often do you play computer games that have vocabulary in English ? (Mark one answer) 
Do not include the time spent on The Forgotten World. 

I I Never 

I | Once every few months 
I | Once or twice a month 
I I Once or twice a week 
I | Three or four times a week 
I I Everyday 

8. Do you go to an English tutoring class outside of school? (Mark one answer) 

I | Never 

I 1 1 have, but not anymore 
I 1 1 do, but have not before 
I 1 1 always have 

9. Compared to when you were in 7 th grade, has your motivation to learn English in 8 th grade: 
(Mark one answer) 

I I Lowered a lot 
I I Lowered a little 
I I Stayed about the same 
I I Improved a little 
I I Improved a lot 

10. How much fun is learning English in school? (Mark one answer) 

I I No fun 
I | A little fun 
I I Some fun 
I | A lot of fun 
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11. Students in general may experience difficulty when they are learning English. Currently, how 
difficult are the following areas to you? (Mark one answer per row) 



Content 


Level of Difficulty 


Not Difficult At 
All 


A Little Difficult 


Somewhat 

Difficult 


Very Difficult 


a. 


Vocabulary 


□ 


□ 


□ 


□ 


b. 


Sentence 


□ 


□ 


□ 


□ 


c. 


Listening 


□ 


□ 


□ 


□ 


d. 


Reading 


□ 


□ 


□ 


□ 


e. 


Writing 


□ 


□ 


□ 


□ 


f. 


Speaking 


□ 


□ 


□ 


□ 



12. Realistically, do you expect to attend university some day? (Mark one answer) 

□ No 
I I Yes 
I I Unsure 

Instruction: Students who have used The Forgotten World, please continue answering 
question 13 -17 below. Students who have never used The Forgotten World do NOT need to 
answer the questions below. 

13. How often did you use The Forgotten World at school ? (Mark one answer) 

I I Never 

I | Once every few months 
I | Once or twice a month 
I I Once or twice a week 
I | Three to four times a week 
I I Everyday 

14. How often did you use The Forgotten World outside of school ? (Mark one answer) 

I I Never 

I | Once every few months 
I | Once or twice a month 
I | Once or twice a week 
I | Three to four times a week 
I I Everyday 
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15. In general, how helpful do you think The Forgotten World is to your English learning? (Mark 
one answer) 

I | Don’t know (Didn’t use much or at all) 

I | Not helpful at all 
I I Not so helpful 
I I Somewhat helpful 
I I Very helpful 

16. What were the two biggest challenges that you encountered while using The Forgotten World? 
(Mark two answers) 

I | Insufficient time for using the software 
I ] Limited support from the school 

I | Problems with computers at the school computer laboratory 
I | Concerns with negative impact on test scores 

I I Other challenges -> Please specify: 

17. What is your username for the Forgotten World? . 



Thank you for completing this questionnaire ! 
We will keep your responses confidential. 
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Table H-1. Hierarchical Linear Models to Confirm Significant Findings for Test 
Score Gains Among Low-performing Students and Improved Motivation 



Hierarchical models of test score change and probability of motivation or fun=l 





Reading score change 


Listening score change 








Model 1 


Model 2 


Model 1 


Model 2 


Spring 

motivation 


Spring fun 




6.61 


6.64 


5.01 


4.59 


-0.43 


-0.24 




(1-05) 


(1.24) 


(0.41) 


(0.63) 


(0.17) 


(0.17) 


Intercept 


0.000 


0.000 


0.000 


0.000 


0.031 


0.189 




1.52 


0.47 


1.12 


-0.83 


0.50 


0.29 




(1.22) 


(1.47) 


(1.31) 


(1.37) 


(0.18) 


( 0 . 19 ) 


Treatment 


0.213 


0.752 


0.391 


0.547 


0.007 


0.122 






-0.04 




0.82 






Below median on 




(0.60) 




(1.04) 






fall test 




0.948 




0.429 






Treatments 




2.24 




4.14 






Below 




(1-03) 




(1.32) 






median 




0.029 




0.002 







Note: Coefficients reported in top line. Standard errors in parentheses. P-values (significance) in third line. 
Significant findings (p<0.05) are bolded and italicized. N=3586 students, 62 classrooms, and 10 schools 



Results are based on hierarchical linear models of change in reading or listening test score, 
predicted by enrollment in a treatment school and, where noted, membership in the lower half of 
the fall test score distribution. These are random-intercept models: the average test score change 
varies by class and school, but the effect of treatment assignment is fixed across all cases. For 
the index of motivation or fun, results are based on hierarchical nonlinear (logistic) models. 
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Table H-2. Mean and Design-corrected Standard Errors for Motivation, for 
Treatment and Comparison Groups 

Data Summary 

Number of Clusters 10 (Schools) 

Number of Observations 3640 



Statistics 

Std Error 

Variable N Mean of Mean 95% CL for Mean 

fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff 

SPRING_Q9_M0TIVATI0N 3567 0.453602 0.028563 0.38898782 0.51821711 

fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff 



Domain Analysis: EXP 

Std Error 

EXP Variable N Mean of Mean 95% CL for Mean 

ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff 

TREATMENT SPRING_Q9_M0TIVATI0N 1787 0.509233 0.017536 0.46956446 0.54890224 

COMPARISON SPRING_Q9_M0TIVATI0N 1780 0.397753 0.040592 0.30592670 0.48957892 

ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff 
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